[gs-bugs] [Bug 698832] - Ghostscript - Undefined behavior in int size test

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=698832

Bug ID: 698832
Summary: Undefined behavior in int size test
Product: Ghostscript
Version: master
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: P4
Component: Test Framework
Assignee: ***@artifex.com
Reporter: ***@mykolab.com
QA Contact: gs-***@ghostscript.com
Word Size: ---

The file `base/gsserrial.h` is a unit test, and the second of the two loops in
its main function is undefined. Shifting into or out of the sign big is
undefined, and that loop intentionally and deterministically does both.

I see that there's a lot of fancy footwork surrounding integers in that test,
so I don't feel confident changing anything. However, remember that because
it's undefined behavior, there's absolutely no guarantee that because it
behaved as expected in one context because it'll behave as-expected in another.

Is anyone here familiar with what exactly these functions are checking? I'm
happy to help develop a clean, standard-compliant fix.

Thanks for your time,
Michael McConville

[b***@artifex.com]

http://bugs.ghostscript.com/show_bug.cgi?id=698832

Ken Sharp <***@artifex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |RESOLVED
Resolution|--- |WONTFIX

--- Comment #1 from Ken Sharp <***@artifex.com> ---
(In reply to Michael McConville from comment #0)

Post by b***@artifex.com
The file `base/gsserrial.h` is a unit test, and the second of the two loops
in its main function is undefined. Shifting into or out of the sign big is
undefined, and that loop intentionally and deterministically does both.

It would be helpful to give a line number or quote the code that you mean.
There aren't actually any loops in the .h file.

Its also kind of hard to see how the .h file has a main function....

The BEGIN and END macros in gsserial.h are there to delimit a macro, they
aren't really loops even though they are defined with a do..while{0}.

There is test code in gsserial.c, so I'm guessing that's what you actually
mean, not the .h file.

I have no idea when that code was last run (if ever) I've certainly not been
aware of anyone running it in the last 10 years. We don't use it in our current
testing scheme and its compiled out. I imagine it was once used to verify the
code in the rest of the function.

However, I think its worth noting that there are *many* places in Ghostscript
where we do things like bit shifting a negative value. Yes, I know its
undefined behaviour and we've removed some of those occurences, but if the
compiler works in an unexpected way then a lot more than the old test code in
gsserial.c is going to fail, and we don't really care about that code anyway
since we aren't using it any more.