[gs-bugs] [Bug 692182] New: xps -> pdfwrite SEGVs with softmasked pattern

[b***@ghostscript.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Summary: xps -> pdfwrite SEGVs with softmasked pattern
Product: GhostXPS
Version: trunk
Platform: PC
OS/Version: All
Status: NEW
Severity: normal
Priority: P4
Component: General
AssignedTo: ***@artifex.com
ReportedBy: ***@artifex.com
QAContact: gs-***@ghostscript.com
Estimated Hours: 0.0


Created an attachment (id=7485)
--> (http://bugs.ghostscript.com/attachment.cgi?id=7485)
fts_30.xps

On the clist_patt_trans branch, I get a SEGV when attempting to do:

xps\debugobj\gxps -o out.pdf -sDEVICE=pdfwrite -r72 -dNOPAUSE -dBATCH
fts_30.xps

It looks like pdev->prev_soft_mask_dict is NULL. This in turn seems to happen
because we run through pdf_exit_substream which pops back up to the previous
'NULL' value.

Given that we set prev_soft_mask_dict to NULL when it's finished with, it seems
reasonable that we should never replace a non-NULL one. If I nobble
pdf_exit_substream to only replace pdev->prev_soft_mask_dict if it's non NULL,
this gets further, but then runs into problems with pdev->subtream_Resources
being NULL.

It seems that we aren't getting the expected nice nesting of:

pdf_enter_substream
pdf_begin_transparency_mask
pdf_make_soft_mask_dict
pdf_end_transparency_mask
pdf_exit_substream

Possibly this change in behaviour is to do with the new clist pattern code used
for patterns with transparency in clist_patt_trans, but I can't see the clist
being used. I will retest on the trunk in a moment.

[b***@ghostscript.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Robin Watts <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|***@artifex.com |***@artifex.com

--- Comment #1 from Robin Watts <***@artifex.com> 2011-05-03 20:19:45 UTC ---
Also crashes on the trunk too.

[b***@ghostscript.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Ken Sharp <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|***@artifex.com |***@artifex.com

--- Comment #2 from Ken Sharp <***@artifex.com> 2012-11-02 09:43:46 UTC ---
For me, on a 32-bit Windows build this does fail (though not as described) in
pdfwrite. However on a 64-bit Windows build it fails (seg fault) reading a PNG
(xps_png_has_alpha calling png_create_read_struct).

The crash in pdfwrite that I can see is in sprintf, which is fairly bizarre as
I can't see any reason for it. Given that the error seems to be moving around I
wonder if this is just a symptom of memory corruption generally.

The 64-bit crash for me also occurs with the command line "gxps -r72
\bugs\692182\fts_30.xps" which clearly doesn't involve pdfwrite at all.

Given that there is a non-pdfwrite problem with memory, and the problem that I
can reproduce with pdfwrite isn't the one Robin describes, I'm passing this one
to the XPS interpreter maintainer.

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Chris Liddell (chrisl) <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Group| |Security
Version|master |unspecified
Priority|P4 |P1
Product|GhostXPS |Security
Component|General |Security
Severity|normal |blocker

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Robin Watts <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #7485|0 |1
is private| |

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Henry Stiles <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|***@artifex.com |***@artifex.com
CC| |***@artifex.com

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Michael Vrhel <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|***@artifex.com |***@artifex.com

--- Comment #3 from Michael Vrhel <***@artifex.com> ---
Ken -- if you could take a second look at this I would appreciate it. I am
only seeing an issue with pdfwrite.

gxdso_pattern_start_accum is called around line 167 in xpstile.c and that is
fine, and then gxdso_pattern_finish_accum is called around line 192 which ends
up in gdevpdfi.c line 2459 calling pdf_add_procsets with
pdev->substream_Resources = NULL which leads to a access violation when we try
to add to a null dictionary.

If after fixing this you are still seeing other issues please hand it back to
me.

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=692182

Michael Vrhel <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|***@artifex.com |***@artifex.com
CC| |***@artifex.com

--- Comment #4 from Michael Vrhel <***@artifex.com> ---
Ken. Sorry for the confusion. I am taking this one back. I now see that
there is some mixup in the softmask that is causing this issue. The
substream_Resources is getting cleared when the soft mask ends. Likely there
needs to be some clean up logic here when things are not nesting nicely. This
file has issues as windows will not open it so it is likely we should error
out.