math: fix Gamma(-171.5) on all platforms

math: fix Gamma(-171.5) on all platforms

Using 387 mode was computing it without underflow to zero,
apparently due to an 80-bit intermediate. Avoid underflow even
with 64-bit floats.

This eliminates the TODOs in the test suite.

Fixes linux-386-387 build and fixes #11441.

Change-Id: I8abaa63bfdf040438a95625d1cb61042f0302473
Reviewed-on: https://go-review.googlesource.com/30540
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>