What if the only thing that existed was your mind, and that everything around you was merely an idea within it? Berkeley (the 18th century philosopher) certainly thought so, but not without good reason. He has many arguments against materialism (that matter exists independently to us), but the best is how it relates to our perceptions:
From George Berkeley's A Treatise Concerning the Principles of Human Understanding (1710):
"All the choir of heaven and furniture of the earth, in a word, all those bodies which compose the mighty frame of the world, do not have any subsistence without a mind--that their being is to be perceived or known, that consequently, so long as they are not actually perceived by me or do not exist in my mind or that of any other created spirit, they must either have no existence at all or else subsist in the mind of some eternal spirit."
So if you agree that we only directly perceive our ideas, then what is it that connects our ideas to objects in the world? How can perceptible qualities such as taste, sight, sound, and touch, exist in an unperceiving substance? All we know about this supposed "matter", understood as existing whether we perceive it or not, are its sensible qualities I mentioned above. There is no reason to believe that these qualities correspond to anything beyond our ideas. In fact, it is a contradiction that matter should exist, for it is a contradiction that sensible qualities exist in things that cannot perceive. And what are physical objects but their constituent sensible qualities? What is a book but it's solid bulk (touch), it's shape and size (touch and sight), its colour (sight), its smell, its sound when you drop it (oops), and its taste (though I hope to never taste a book, thank you very much). To look at this another way, you don't have a physical substance existing unperceived any more than you have taste floating about without something to taste it.
Matter is often known as "the great illusion", in both philosophy and mysticism, and this ties in well with Berkeley's idealist view of matter. Matter is indeed less real than we often believe, and what is more real and permanent is the mind that perceives it. This could mean that we have more control over the world than we might imagine. Some of our thoughts are easier to manipulate than others, such as our imagination, our goals, etc, as opposed to what we believe exists "out there" in the world. But if our thoughts about the world (about the "matter" out there) and those that we normally consider as more personal to us (imagination, goals, musings) are on par with respect to their metaphysical status, then perhaps those thoughts about matter can also be changed just like we can change our beliefs (not that it is always easy to change our beliefs, but we all know it can be done). So what is the world, then, or perhaps, whose world is it? Whose thoughts are you living in? Are they your own, or someone else's?
Another thing Berkeley says is "What bickerings and controversies, what a learned dust has been raised about those matters..." I laughed out loud when I first read this: I imagined all these old British philosophers muddling about in a dust storm in the desert. Muddling about when they don't realize it's all in their minds. So next time, when someone says, "Don't worry, it's just in your mind", you can happily agree, and say, "Yes, and you are too."
UPDATE (May 25, 2015): I was just reading something about quantum mechanics (QM) that reminded me of this. In QM, a particle can be thought to not actually exist until it is observed (which can be that a measurement is made of its momentum or position). Well, this fits perfectly with idealism, (ignoring God, the eternal observer, for the moment), since an object cannot be said to exist unless it is observed and becomes a thought in our minds. So if objects don't exist without us (or any observer), they are not mind-independent, and so there is no matter "out there" just like Berkeley believed.
Just thought that was worth adding :)