Xorg fails to start with radeon (kernel rejects radeon command stream from Xorg)

[Jean-Baptiste Lallement]

The shell fails to start with latest version of xorg and radeon drivers.

From the journal:

sept. 06 08:06:26 sark /usr/lib/gdm3/gdm-x-session[6404]: (II) RADEON(0): Setting screen physical size to 508 x 285
sept. 06 08:06:26 sark /usr/lib/gdm3/gdm-x-session[6404]: radeon: The kernel rejected CS, see dmesg for more information (-22).
sept. 06 08:06:26 sark kernel: radeon 0000:01:00.0: evergreen_cs_track_validate_texture:855 texture bo too small (layer size 8847360, offset 0, max layer 1, depth 1, bo size 4096) (1920 1152)
sept. 06 08:06:26 sark kernel: [drm:radeon_cs_ioctl [radeon]] *ERROR* Invalid command stream !

The shell starts normally under wayland.
Downgrading the following packages fixed the problem:

xserver-xorg-video-qxl:amd64 (0.1.5-2build2, 0.1.5-2build1)
xserver-xorg-video-vesa:amd64 (1:2.4.0-1, 1:2.3.4-1build3)
xserver-xorg-video-amdgpu:amd64 (18.0.1-1build1, 18.0.1-1)
xserver-xorg-core:amd64 (2:1.20.1-1ubuntu1, 2:1.19.6-1ubuntu4)
xserver-xorg-video-fbdev:amd64 (1:0.5.0-1, 1:0.4.4-1build6)
xserver-xorg-video-intel:amd64 (2:2.99.917+git20171229-1build1, 2:2.99.917+git20171229-1)
xserver-xorg-video-vmware:amd64 (1:13.3.0-2build1, 1:13.2.1-1build1)
xserver-xorg-video-ati:amd64 (1:18.0.1-1build1, 1:18.0.1-1)
xserver-xorg-video-radeon:amd64 (1:18.0.1-1build1, 1:18.0.1-1)

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: xserver-xorg 1:7.7+19ubuntu8
ProcVersionSignature: Ubuntu 4.17.0-9.10-generic 4.17.17
Uname: Linux 4.17.0-9-generic x86_64
ApportVersion: 2.20.10-0ubuntu9
Architecture: amd64
CompizPlugins: [core,composite,opengl,compiztoolbox,decor,vpswitch,snap,mousepoll,resize,place,move,wall,grid,regex,imgpng,session,gnomecompat,animation,fade,unitymtgrabhandles,workarounds,scale,expo,ezoom,unityshell]
CompositorRunning: None
CurrentDesktop: ubuntu:GNOME
Date: Thu Sep 6 15:03:11 2018
DistUpgraded: 2018-03-24 15:02:43,050 DEBUG Running PostInstallScript: './xorg_fix_proprietary.py'
DistroCodename: cosmic
DistroVariant: ubuntu
ExtraDebuggingInterest: Yes
GraphicsCard:
 Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770] [1002:68b8] (prog-if 00 [VGA controller])
   Subsystem: PC Partner Limited / Sapphire Technology Juniper XT [Radeon HD 5770] [174b:1482]
InstallationDate: Installed on 2014-07-15 (1514 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Alpha amd64 (20140520)
MachineType: Gigabyte Technology Co., Ltd. GA-890GPA-UD3H
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.17.0-9-generic root=UUID=0889d734-ea91-4bdb-9d16-2a0a923ad3d1 ro quiet splash vt.handoff=1
SourcePackage: xorg
UpgradeStatus: Upgraded to cosmic on 2018-03-24 (165 days ago)
dmi.bios.date: 07/23/2010
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: FD
dmi.board.name: GA-890GPA-UD3H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrFD:bd07/23/2010:svnGigabyteTechnologyCo.,Ltd.:pnGA-890GPA-UD3H:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnGA-890GPA-UD3H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: GA-890GPA-UD3H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.
version.compiz: compiz N/A
version.libdrm2: libdrm2 2.4.93-1
version.libgl1-mesa-dri: libgl1-mesa-dri 18.1.5-1ubuntu1
version.libgl1-mesa-glx: libgl1-mesa-glx 18.1.5-1ubuntu1
version.xserver-xorg-core: xserver-xorg-core 2:1.19.6-1ubuntu4
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev 1:2.10.6-1
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:18.0.1-1
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.917+git20171229-1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau N/A

[Jean-Baptiste Lallement (jibel)]

Journal from a failed boot.

[Daniel van Vugt (vanvugt)]

Hmm, your monitor is 1920x1080 and yet the kernel says "texture bo too small" with "1920 1152":

sept. 06 08:06:26 sark kernel: radeon 0000:01:00.0: evergreen_cs_track_validate_texture:855 texture bo too small (layer size 8847360, offset 0, max layer 1, depth 1, bo size 4096) (1920 1152)

It sounds like a texture that's 1152 tall would actually be too big, not too small.

Regardless, this seems to confirm there's a bug/disagreement somewhere between Xorg and the kernel.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Daniel van Vugt (vanvugt)]

Looking at the kernel source (drivers/gpu/drm/radeon/evergreen_cs.c), it appears the value:

  surf.nby == 1152

is wrong.

Since the layer size is 8847360 bytes and the correct height 1080 pixels, this means you have a stride of 8192 bytes per row (which is 1920 pixels rounded up to 2048, multiplied by 4 bytes per pixel). So in the least the error message looks buggy: It should not say "1920 1152". But the trigger for the error also isn't directly related to that...

The trigger for the error is:

        /* toffset == texdw[2] << 8 == 0 */
        /* depth = 1 */
        /* mslice == 1 */
        /* surf.layer_size == 8847360 */

        if (dim == SQ_TEX_DIM_3D) {
                toffset += surf.layer_size * depth;
        } else {
                toffset += surf.layer_size * mslice;
        }

        /* toffset == 8847360 */
        /* radeon_bo_size(texture) == 4096 */

        if (toffset > radeon_bo_size(texture)) {
                dev_warn(p->dev, "%s:%d texture bo too small (layer size %d, "
                         "offset %ld, max layer %d, depth %d, bo size %ld) (%d %d)\n",
                         __func__, __LINE__, surf.layer_size,
                        (unsigned long)texdw[2] << 8, mslice,
                        depth, radeon_bo_size(texture),
                        surf.nbx, surf.nby);
                return -EINVAL;
        }

Those are two very different values. Of course toffset is bigger. I can't tell if that's a mistake in the kernel logic or an encoding mistake from mesa (or Xorg).

[Timo Aaltonen (tjaalton)]

there's an update to x-x-v-ati which got stuck in -proposed, maybe try that out? It carries a bunch of cherry-picks from upstream to fix glamor issues with current xserver. There's also a new upstream release which I'll push to Debian in a bit.

[Jean-Baptiste Lallement (jibel)]

This is fixed with x-x-v-ati 1:18.1.0-1

Thanks!