There are two hypotheses as to why people see dramatically different things; one is that our brains are constantly being bombarded by information and so we end up making a lot of assumptions and interpretations based on parameters. If you are looking up close at something and infer a blue background, you see the dress differently than people who assume it has an artificial light background, like yellow.
Then there is color constancy. Our retinas use rods and cones which contain a visual pigment embedded in the cell membrane of the photoreceptor to see color or black and white. Rods are sensitive to black, white and shades of gray while the three types of cones are sensitive to red, green and blue. But cones do not work as well in low light so things with varying brightness look different.(1)
Look at the cube on the left, courtesy of AsapSCIENCE.(2) The shadow makes the color of two identical blocks look like different colors.
Blue cones are much less sensitive to light, you need twice as much to get the same response as with red and green. So light matters in looking at the blue dress. What is the light source for everyone looking at the dress? A tablet or a PC monitor, all of which have varying levels of brightness being viewed in variously lit rooms on various quality screens. If we were all in the same room using the same light, it would look the same color, but not on tablets and monitors.
It was only a matter of time before someone started making value judgments on the whole dress thing - no, it was not Mother Jones insisting that super-smart liberals see one color while evil Republicans see another (though that is probably in the works), instead it was humanities people elitely lamenting that anyone cared about this.
In actuality, a lot of people got a lot smarter about eyes and color perception because of this. Perhaps not everyone, some people always learn just enough to be wrong, like Mother Jones when it comes to science. In film, learning some color theory is why Michael Bay movies can be so painful to watch - he learned that in color theory, gold and teal are opposites and so will draw the eye's attention to whatever is between them. In 2009's "Transformers: Revenge of the Fallen", he filmed the entire movie hammering everything with those two colors, which meant he was constantly pulling on our brains.
The real mind blower is how other creatures might see that dress. While we have 3 visual pigments plus black and white, a mantis shrimp has 16. They are not just seeing red, green and blue, they can perceive polarized light and multispectral images and can detect five different frequencies just in deep ultraviolet. To a mantis shrimp, that blue dress could be really blue.
The only creature that can truly see a mantis shrimp in all its glory is another mantis shrimp.