I've said this hundreds of times, and I'll say it one more time. Before spending money on equipment, spend a little bit of time (and money) on education. Get a copy of John Shaw's Nature Photography Field Guide.
All cameras that I am about to recommend have RAW mode capability. What this means is that the camera is capable of capturing all the data picked up by the sensor and dumping it into storage. This is like shooting a color negative: a lot of information is captured, more than can be shown on a computer screen, so you'll need software to process the output from the digital sensor into something that can be shown on screen. I use Adobe Photoshop Lightroom 3, which is designed for a photographer's workflow. I cannot recommend free software such as Picasa or iPhoto. Picasa's workflow was designed around JPG, and bogs down tremendously on RAW files. iPhoto has scalability problems. Neither have the tools that Lightroom has for rescuing badly over-exposed photos, adding in ND Grad filters after the fact, or a reasonable tagging/captioning system. The software pipeline determines the final quality of the image, and is responsible for your ability to shoot whatever you want in various different lighting conditions without having to carry a bunch of different filters with you, so it's important not to skimp on this.
Point and Shoots
Everyone knows what a point and shoot camera is. My currently recommended point and shoot for serious photography is the Canon PowerShot S95. The important thing that separates this camera from its competitors is the user interface. Rather than clumsy buttons that have to be pushed/held/pushed, the S95 has 2 dials that let you adjust aperture and control for exposure compensation at the same time. In tricky situations where you want manual exposure controls this is the camera to beat. As a pocket sized camera it's easily carried wherever I go, and now that everyone else has switched to phone cameras, photos from the S95 look nothing short of outstanding. You can read my full blown review of the S90, which was the S95's predecessor.
Interchangeable Lens Rangefinder
The big limitation of a point and shoot is sensor size. Sensor size matters a lot. The optics and the physics involved is such that the larger sensors always outperform the smaller ones. If you size up the sensor in a point and shoot, however, you have to size up the lens, and so the whole camera becomes bigger. To get the most out of the improved sensor, however, you should stick a better lens in front of it. Since the physics of lenses is such that you cannot design a one-size-fits-all lens that performs worth a damn, it makes sense to build an interchangeable lens mount so you can put different lenses on the camera depending on what you want to shoot. (For you disappointed SLR owners, yes, that means the 18-200mm lens you've slapped onto your body is under-performing the cheap $300 24-85/3.5-4.5 lens I bought 10 years ago and sold to another googler for $150)
The prototypical camera for this class of cameras is the Olympus PEN E-PL1. The Micro Four-Thirds standard is supported by Olympus and Panasonic so you can buy interchangeable lenses from either manufacturer for this standard and it will work. You can see that the PEN is about the same size as the Canon G12, but will outperform it in all conditions by a substantial margin because of the larger sensor size and better lens. For the price difference it is a no brainer to go with the Micro Four Thirds cameras. Sony also makes the Sony Alpha NEX NEX5A/B Digital Camera which is of the same class. I've never so much as touched one but Pengtoh raves about it.
An SLR is a Single-Lens-Reflex camera. The "reflex" refers to the mirror that sits in the optical path, reflecting light coming in from the lens into the penta-prism which then gives you an upright view through a view-finder. When you press the shutter release, the mirror flips up (the reflex action) and the optical path switches to exposing the sensor to the incoming input. The "single" part is in contrast to Twin-Lens-Reflex cameras, which are an obsolete design that nobody really cares about any more. Why is the mirror important? The previous two classes of cameras do not have a purely optical path for you to view the image before shooting. That means that you are generally restricted to viewing the image on a relatively low-resolution LCD screen. That does two things. First, in bright light the screen becomes useless and you are now effectively shooting blind. That's acceptable if you don't care about the results, but the whole point of carrying anything but a cheapo point and shoot is that you do care about the results, and the viewfinder/mirror optical path solves that problem. Secondly, the viewfinder, by presenting a high resolution image (better than 1080p!) lets you do precise filter placement so you can see exactly what the filter does.
For instance, when shooting a rainbow, what you want to do is to use a polarizer so that you can eliminate the specular reflections from it. This makes the rainbow appear stronger and is far closer to what your eyes see than an unfiltered image:
|From 2010 Canadian Rockies Fall Colors|
You can try applying a filter to an interchangeable rangefinder camera, but you will find it difficult to judge the results on the screen. Point and shoots don't even take filters without a lot of work.
For seriously tricky light when ND grad filters are a necessity, nothing short of an SLR would be useful. Precise placement of the ND grad filter can only be accomplished with the lens stopped down to shooting aperture, and an optical path with which to see the image. That's why I don't own an interchangeable rangefinder camera: there's no situation which a rangefinder works where a point and shoot wouldn't do an equivalent job.
Note that DSLRs come in 2 classes: full frame sensor cameras and crop sensor cameras. The full frame sensor cameras use a sensor the size of a traditional 35mm film. The crop sensor cameras use a sensor that's about half the size. Since size matters as far as image quality is concerned, my approach was to wait until full frame sensor cameras reached sufficient quality for me to switch from film to digital. That happened in December 2008 when the Canon EOS 5D Mark II was released. I personally can't be bothered to carry an SLR that's anything short of full frame (after all, why carry a full 35mm lens and use only half of it?!!), but if you're not as fanatical about image quality as I am, the crop sensor cameras can be a good choice.
ConclusionThere are other types of cameras not covered above. View cameras, for instance, are a classic landscape photographer's tool. But very few people are willing to carry such cameras, and as someone who enjoys traveling fast and light, even the DSLRs are sometimes too cumbersome.
The sad thing about all this is that most people have gone to crappy phone cameras, none of which will capture in RAW mode, or even produce sharp pictures because the unprotected lenses are usually smudged with fingerprints and the auto-focus mechanisms suck. The nice thing though, is if you actually pay attention, learn a little bit about photography, and even carry the cheapest camera in this article, then you stand a good chance of producing photos that will wow your audience who have been trained to expect crap pictures.
Regardless, you should pick the camera to suit your purpose. If all you do is shoot baby pictures, get a point and shoot or an interchangeable lens rangefinder. If you want to play tripod artist in the dying light of the day and are capable of achieving technical mastery over your equipment, then a digital SLR will let you express your creativity and vision without placing limitations on what you can or cannot do.
|From 2010 Canadian Rockies Fall Colors|
For more information, see Philip Greenspun's on-line textbook on Photography