With them having 3% spandex would you recommend going down a size or not?

I wear a 31, and I wish I would have bought a 30. I bought a 32, and they will expand a little as you wear them, as I should have expected. THey stay pretty in-size, but I don't wash my pants super often, to keep them durable. Nothing a belt can't fix, but definitely try a size below - they'll stretch a bit but stay to fit.