[gs-bugs] [Bug 697483] - Ghostscript - Segfault with psdcmyk using -dGraphicsAlphaBits=4 -dTextAlphaBits=4 Altona_Technical

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697483

Bug ID: 697483
Summary: Segfault with psdcmyk using -dGraphicsAlphaBits=4
-dTextAlphaBits=4 Altona_Technical
Product: Ghostscript
Version: master
Hardware: PC
OS: Windows NT
Status: UNCONFIRMED
Severity: normal
Priority: P4
Component: Graphics Library
Assignee: ghostpdl-***@artifex.com
Reporter: ***@artifex.com
QA Contact: gs-***@ghostscript.com
Word Size: ---

This turned up on the nightly run 1/15 and I am able to duplicate it on
Windows using:

gswin64c -sDEVICE=psdcmyk -o x-%d.psd -dTextAlphaBits=4 -dGraphicsAlphaBits=4 \
-r300 -Z: tests_private/comparefiles/Altona_Technical_v20_x4.pdf

Since I observed the segfault on page 7, I determined that it happens very
quickly if I add -dFirstPage=7

With a debug build I see the segfault is in i_free_object when called from
gx_default_copy_alpha when trying to free "lout". This is caused by the
earlier call in line 395 to:

code = (*dev_proc(dev, get_bits)) (dev, ry, lin, &line);

which writes past the end of "lin" area overwriting the buffer allocated for
"lout".

This happens on the very first call to gx_default_copy_alpha. Note at this
point the 'dev' is the pattern_accumulator and its target is image24, but
both the pattern_accumulator device and the image24 device have is_planar set
to 1 and the get_bits_rectangle directs to mem_planar_get_bits_rectangle.

The problem is that "in_size" from gx_device_raster for planar devices is for
a plane, not for all planes, so when the mem_planar_get_bits_rectangle returns
3-bytes per pixel, it overflows (by a lot).

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697483

Ken Sharp <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|ghostpdl-***@artifex.com |***@artifex.com
Priority|P4 |P1

--- Comment #1 from Ken Sharp <***@artifex.com> ---
This one still exhibits. I'm not sure who should own it, I'm going to assign it
to Ray for now as he reported it, but it probably needs to be reassigned.

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697483

Chris Liddell (chrisl) <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Group| |Security
Version|master |unspecified
Product|Ghostscript |Security
Severity|normal |blocker
Component|Graphics Library |Security

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697483

Ray Johnston <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|UNCONFIRMED |RESOLVED

--- Comment #2 from Ray Johnston <***@artifex.com> ---
This no longer fails. Bisecting shows that the SEGV was cured by:

commit 143a892226b79676f5f8853f63532f9e6f4e8f5e
Author: Robin Watts <***@artifex.com>
Date: Mon Nov 13 11:05:04 2017 +0000

Avoid buffer overflow in GraphicsAlphaBits cdoe with planar device.

In GraphicsAlphaBits, we get the pixels from the underlying device
(in chunky format). The buffer we get that data into was sized
according to get_device_raster(dev, false), which (for a planar
device) returns the size for line of planar data. We want the
size for one line of chunky data, so use get_device_raster_chunky
instead.

Note many pages still get errors due to invocation of
gx_default_copy_alpha_hl_color