[gs-bugs] [Bug 697536] - Ghostscript - constrained memory crash

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697536

Bug ID: 697536
Summary: constrained memory crash
Product: Ghostscript
Version: unspecified
Hardware: PC
OS: Linux
Status: UNCONFIRMED
Severity: normal
Priority: P4
Component: Graphics Library
Assignee: ghostpdl-***@artifex.com
Reporter: ***@artifex.com
QA Contact: gs-***@ghostscript.com
Word Size: ---

Crash during band playback:

valgrind --track-origins=yes ./gpcl6 -dUseFastColor=true -K32000 -r1200
-sDEVICE=pbmraw -o /dev/null JEITA/J11.pxl
==5933== Memcheck, a memory error detector
==5933== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==5933== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==5933== Command: ./gpcl6 -dUseFastColor=true -K32000 -r1200 -sDEVICE=pbmraw -o
/dev/null /home/henrys/cluster/tests_private/xl/JEITA/J11.pxl
==5933==
[a+]gs_malloc(alloc_ht_cache(bits))(541440) = 0x0: exceeded limit,
used=32451400, max=32451400
==5933== Invalid read of size 8
==5933== at 0x6AF029: chunk_free_object (gsmchunk.c:632)
==5933== by 0x8E76B2: gx_ht_order_release (gsht.c:574)
==5933== by 0x6AAAEA: gx_ht_read_and_install (gxdhtserial.c:598)
==5933== by 0x683915: read_ht_segment (gxclrast.c:2591)
==5933== by 0x67FF50: clist_playback_band (gxclrast.c:1745)
==5933== by 0x68925B: clist_playback_file_bands (gxclread.c:917)
==5933== by 0x688F40: clist_render_rectangle (gxclread.c:851)
==5933== by 0x688A2B: clist_rasterize_lines (gxclread.c:740)
==5933== by 0x6883EE: clist_get_bits_rectangle (gxclread.c:629)
==5933== by 0x6AD485: clist_get_bits_rect_mt (gxclthrd.c:754)
==5933== by 0xA2C1DF: gx_default_get_bits (gdevdgbr.c:54)
==5933== by 0x662368: gdev_prn_get_bits (gdevprn.c:1641)
==5933== Address 0xf1f1f1f1f1f1f221 is not stack'd, malloc'd or (recently)
free'd
==5933==
==5933==
==5933== Process terminating with default action of signal 11 (SIGSEGV)
==5933== General Protection Fault
==5933== at 0x6AF029: chunk_free_object (gsmchunk.c:632)
==5933== by 0x8E76B2: gx_ht_order_release (gsht.c:574)
==5933== by 0x6AAAEA: gx_ht_read_and_install (gxdhtserial.c:598)
==5933== by 0x683915: read_ht_segment (gxclrast.c:2591)
==5933== by 0x67FF50: clist_playback_band (gxclrast.c:1745)
==5933== by 0x68925B: clist_playback_file_bands (gxclread.c:917)
==5933== by 0x688F40: clist_render_rectangle (gxclread.c:851)
==5933== by 0x688A2B: clist_rasterize_lines (gxclread.c:740)
==5933== by 0x6883EE: clist_get_bits_rectangle (gxclread.c:629)
==5933== by 0x6AD485: clist_get_bits_rect_mt (gxclthrd.c:754)
==5933== by 0xA2C1DF: gx_default_get_bits (gdevdgbr.c:54)
==5933== by 0x662368: gdev_prn_get_bits (gdevprn.c:1641)
==5933==
==5933== HEAP SUMMARY:
==5933== in use at exit: 32,551,940 bytes in 944 blocks
==5933== total heap usage: 3,546 allocs, 2,602 frees, 60,748,266 bytes
allocated
==5933==
==5933== LEAK SUMMARY:
==5933== definitely lost: 0 bytes in 0 blocks
==5933== indirectly lost: 0 bytes in 0 blocks
==5933== possibly lost: 0 bytes in 0 blocks
==5933== still reachable: 32,551,940 bytes in 944 blocks
==5933== suppressed: 0 bytes in 0 blocks
==5933== Rerun with --leak-check=full to see details of leaked memory
==5933==
==5933== For counts of detected and suppressed errors, rerun with: -v
==5933== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697536

Henry Stiles <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|ghostpdl-***@artifex.com |***@artifex.com

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697536

Chris Liddell (chrisl) <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Group| |Security
Priority|P4 |P1
Product|Ghostscript |Security
Severity|normal |blocker
Component|Graphics Library |Security

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697536

Ray Johnston <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Ever confirmed|0 |1
Status|UNCONFIRMED |IN_PROGRESS

--- Comment #1 from Ray Johnston <***@artifex.com> ---
This no longer occurs. On Windows it runs until it gets a -992 processing an
image (change compression in an image).

On linux, under valgrind, it dies with:
Warning interpreter exited with error code -998
Flushing to end of job
PCL XL error
Subsystem: KERNEL
Error: InsufficientMemory
Operator: BeginPage
Position: 14061
file position of error = 723817204

which seems reasonable with the -K32000

Note that with a debug build, I also get:
[a+]gs_malloc(alloc_ht_cache(bits))(541344) = 0x0: exceeded limit,
used=32227736, max=32227736
[a+]gs_malloc(alloc_ht_cache(ht_tiles))(236856) = 0x0: exceeded limit,
used=32736096, max=32736096
Warning interpreter exited with error code -998
Flushing to end of job

(this indicates that the cited results were from a debug build, since only
that gives the [a+] messages.

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=697536

Ray Johnston <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|IN_PROGRESS |RESOLVED

--- Comment #2 from Ray Johnston <***@artifex.com> ---
git bisect indicates that this was fixed by:
commit 7bb2ad6cce75aae3ebedaa5883e1ecd6ba19b52d
Author: Shailesh Mistry <***@hotmail.co.uk>
Date: Thu Feb 16 07:56:26 2017 +0000

Bug 697545: Memory Squeezing fix.

Fix for 'red 85546'. This is a SEGV seen at allocation event
85546 when memory squeezing:

gs -sDEVICE=bit -o /dev/null: examples/tiger.eps

Also account for all the other places that allocate a new
colorspace without checking the returned value.