In the strictest physical sense, only the altitude of the ground and the physical obstacles upon it would limit the lowest altitude at which a plane can fly. However, there are a wide array of procedural constraints and legal requirements that make the answer to this question more interesting. After all, neither airliners nor small planes are permitted to cruise along at treetop level over populated areas. Indeed, numerous pilots have gotten into serious trouble for "buzzing" someone's house.
The first and most obvious limit to how low airplanes can fly is the ground. In aviation, there are two standards for measuring altitude: height above sea level and height above the ground. Height above sea level establishes an unchanging standard, which simplifies communications between pilots and controllers, as well as pilots and other pilots. However, there is one problem with this standard: the elevation of the ground above sea level changes. 2,000 feet above sea level is a fine altitude for flying in a coastal area where the ground is near sea level. However, when flying in the mountains, 2,000 feet could be deadly. Therefore, aviation maps have indications of the terrain elevation on them, and also have indications of the minimum safe altitude. Staying at or above this minimum altitude makes it easy for pilots to stay clear of both terrain and obstacles, even if the weather is cloudy and the ground is not visible.
There are two times when airplanes normally fly at extremely low altitudes: the first few minutes after takeoff and the last few minutes before landing. These are the most critical times in any flight, since there is less margin for error than usual. The airspace around controlled airports is strictly regulated, and planes approaching and separating the runways are required to meet strict altitude requirements at specific geographical fixes. This become even more critical if the airports is close to mountains, hills, tall buildings or other potential obstacles.
In additions to the danger posed by terrain and obstacles, flying low near the ground has another major disadvantage as well, especially for jet powered aircraft. The air close to the ground is significantly denser than the air at higher altitudes. This means that the engines must work harder to push the aircraft through the air at low altitudes. This is the primary reason that airliners generally cruise at 30,000 feet or higher, since these altitudes offer much better fuel efficiency. One other consideration is visibility. In cloudy weather, flying close to the ground become even more dangerous. Therefore, pilots are required to adhere to a stricter set of rules in poor visibility.
It is a common misconception among people unfamiliar with aviation that it is safer to fly low and slow. In fact, low and slow is the most vulnerable and dangerous situation an airplane can be in. At low speeds, airplanes become unstable and are in danger of falling out of the air. At low altitude, there is very little room to maneuver. This is the usual configuration of an airplane on approach to land, and is the reason that landing is one of the most difficult skills for a pilot to learn.
In the earliest days of aviation, most airplanes has weak motors and relatively bulky, unstreamlined fuselages. Therefore they could not fly very high or very fast. In fact, the original Wright brothers Flyer only flew a few feet off the ground for a few seconds at a time. By World War I, airplane were only moderately more sophisticated, and most plane flew no more than a few thousand feet above the ground. This is one of the reasons that flying was so dangerous then. The airplane truly came into its prime in World War II. By then, many fighters and bombers regularly flew at heights of 20,000 feet or more.