EDIT 7/4/12: Added a couple of additional sources for wool.
I'll probably add more to this later. This is various bits of info about needle felting, and my favorite shops for supplies. They're all in the US though, I really don't know where to get supplies outside the US. Some will ship internationally, though.
Here's what I'd recommend to get started felting: The most important tool would be a good felting needle. I haven't seen the better needles or good wool in craft stores, so you'll have to go to a specialty store or order online. I got some needles from Michael's because they were cheap, but they don't have very many barbs so I don't generally use them now. "Star" needles are my favorite, because they have more sides and more barbs to catch the wool. You don't really need multiple sizes to get started, but you might get at least one spare needle. They do break sometimes. (I'm special, I hit my wire armatures all the time with no problem, but I've broken all sizes of needles on eye details, and a dryer ball.) There are many sizes (gauges) and blade styles of needles, but in the beginning you only need the more common ones: A 38 gauge star (38S) is excellent for practically everything. If you only get one type, get a 36 or 38 star. A 36 triangle (36T) can be good for the initial sculpting of core wool, as it's larger. The most common fine gauge I see is 40T, these are good for doing tiny details, such as eye highlights. If you really get into felting, a "fancy" needle that I like is the crown tip. It has barbs only at the tip of the needle, so it felts only the surface fibers. That's normally not desirable, and some people have trouble avoiding that with regular needles, but it does have a use: Flat shapes can be made more easily without poking through to the other side, for instance. I use crowns for making figures posable more easily, as my tendency is actually to felt too hard! By using a crown needle on a figure's legs or tail, I leave the wool near the wire unfelted, so it's soft enough to bend. Be careful about buying specialty needles though: since they're usually only available in bulk, some sellers either try to quickly recoup the expense or are just mean and overcharge for them. Star blades and specialty styles usually cost more, but you should never pay more than $2 per needle. I get specialty needles on eBay, you can often find crowns sold for "reborn" doll hair rooting. Just make sure to check the picture, I have seen standard barbed needles labeled as crowns at least once!
For wool, try to get "batt" or "batting." The hairs are mixed up so they felt more quickly, and they're also often more coarse in size. (Generally, the finer the fiber the more time it takes to felt) Here's a video that explains the different processing levels, I found it helpful. It does simplify a bit, though. For instance, the actual difference between sliver and top is that top is combed rather than carded, but it's not important to know that in the beginning. You can get "core wool" cheaply, this is batt that isn't processed much and felts really easily. It's cheap because it's undyed, and too coarse for other fiber crafts. It hasn't been carded very much so it still has little bits of grass in it. It is washed though, so it's clean. You can also get colored wool in batt form, if you don't want to make a core first. I found it easier to start out with dyed batts without making a core for my shape, but that's because I have a hard time envisioning how to shape something out of two different materials. (I can't sculpt clay around a foil armature for that reason) After I made a few things, I started using the core wool. Roving, sliver, and top will felt pretty well too, but not as easily as batt so I prefer to leave those for the "top coat" only. They stick to the batt core easily, but not so well to themselves. Stay away from synthetics at first, they usually take more effort to felt. If I had tried to felt poly-fil first, like some people do, I would have gotten frustrated and given up because it takes so long. Wool is the most common fiber available, but any animal fiber will felt, as will silk. (Fun fact: The luxury fiber chiengora is the undercoat of double coated dogs!) Alpaca and angora (as in the rabbit breed, not the goat. Angora goats produce "mohair.") are fairly common as well, though a bit more expensive. Expect to pay around $2.50 per ounce of high quality carded or combed fiber from sheep, with other animals being more.
Another thing that is very helpful, though not always necessary, is a foam block. This acts as a work surface and helps protect against stabbing yourself. I spend roughly half my felting time using the foam, though it varies with the project. You use a foam block when felting flat pieces, and very small ones. It's also handy to stick your needles in it when you're not using them. (My block has a ton of needles stuck in it, because I use different styles and sizes, and some multi needle tools) This is another thing that gets overpriced sometimes. Any high density foam that's at least a couple inches thick will do. Many artists use upholstery foam, which is easily found at craft stores. I've found it at Wal-Mart too. I prefer a firmer surface, so currently I'm using a block of very stiff packing foam from a shipping box of electronics. Living Felt sells decently priced foam blocks. They're very dense and durable. I have one of the smaller ones, but my packing foam block is bigger so I've been using that. Just make sure the foam you get is flexible - styrofoam won't work! It's too stiff and could break your needles. You want some give.
I got my first good materials from Living Felt. They have a nice selection and good prices. They even have kits for making bears and stuff, and a starter kit. I've gotten a lot of my needles from here.
I've also gotten some really nice wool from New England Felting Supply. I got some of their superfine merino batt, and core wool ("People's Wool"). The short fiber merino is much softer than the top I got from Living Felt because it's finer, and the core wool was cleaner.
Paradise Fibers sells a lot of interesting, less common fibers, like pure angora and bamboo. The only downside is you have to order in bulk, there's a quantity limit on everything. Usually 4 ounces is the minimum. They also sell the more common needles.
I get Huacaya and Suri alpaca fiber from Breezy Ridge Alpacas, NorthStar Alpacas, and The Critter Ranch. They're all excellent to deal with, have quality fleece, and great prices.
The main producer of needles in the US is Foster Needle. You can buy direct, but as I mentioned before it's only in very large quantities (1000 pieces usually), so there's not much point in that unless you're going to resell. But the site is very informative on the different needle styles, and has a lot of diagrams.
Felt Alive is a big supplier in the US, along with Living Felt. I haven't bought from them yet, but they have a lot of cool stuff, including some less common needle styles. They also have a lot of starter kits, most of which include a unique form of joint construction. The owner is also on deviantART.
A good medium-fine wool that's cheap is this unnamed wool from The Felt Store. You can also buy it on eBay and The Felt Store's own website. It's bright white and very soft, though not as soft as superfine. It has a nice crimp, making it a good choice for dress fiber that isn't too hard to felt. imaginaryfriends2012 has used it in many of her figures.
Magic Cabin's wool stuffing is a cheap choice for core wool, though I personally don't care for it. It's too coarse and hairy for the small scale I usually work at. I do know some needle felters like it, though. It's good for bulking out a large piece that doesn't need detail.
As for technique, I learned mostly by doing. There are tons of free tutorials online though, including on dA. The one that got me to actually try felting was this one, because it's so small and simple: How to make Mr. Turtle The only book I have is Needle Felting: To The Point which is a great resource and was helpful to me when I got confused by a few things. She also has another book, but I haven't gotten it yet. This video was also helpful to me, it shows making simple figures with armatures. Sometimes video is better than stills!