cmd/internal/obj/ppc64: refactor maskgen

cmd/internal/obj/ppc64: refactor maskgen

Refactor maskgen into decodeMask32. This is derived from
from combining encodePPC64RotateMask and isWordRotateMask.

Also, truncate me returned from decodeMask32/64 to
be within range of [0,32/64).

Change-Id: Ie9efff93d400b3066ac85276b1ad3c57c2fcf31b
Reviewed-on: https://go-review.googlesource.com/c/go/+/536298
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Lynn Boger <laboger@linux.vnet.ibm.com>
Run-TryBot: Paul Murphy <murp@ibm.com>
Reviewed-by: Than McIntosh <thanm@google.com>