The Shrub Identification Book by George W.D. Symonds states that brown pith separates Swamp from Red-Osier Dogwood and I think just helps cement this ID. I have found some web documentation that ties brown pith to a different dogwood species but I will stick with my ID book here.
The leaves do take on slightly different shapes.
The Shrub Identification Book also illustrated the flowering, saying that a flat topped cluster of flowers, as opposed to conical, indicates a few species that include Swamp Dogwood. Also noteworthy, all Dogwood flowers have four petals.
So that's it! Lots of bits and pieces that come together to strongly indicate Swamp Dogwood. As I mentioned, there is conflicting species identification properties and as well even some possible naming changes over the years. Regardless, I will stick with this ID for now, but that may change once this book comes in. More to come! :-)