Can you imagine a consumer product that mixed the nastiest of chemicals with perfumed scents and then sprayed the combination throughout your home and workplace in a finely aerosolized concentration for maximum penetration into the lungs and sinuses? Well, next time the Little Woman breaks out the air freshener, run and hide because that is exactly what is happening in almost every case. Remember: when it comes to most consumer products, a little madness is unfortunately the norm. Again, who can miss the irony is that many moms and businesses, trying to make their surroundings smell more pleasant, are actually putting those around them at considerable risk.
How bad is the issue? The concern with air fresheners are not only the products themselves, such as VOC's (Volatile Organic Compounds) and terpenes, but also the byproducts produced when they hit ozone in home air. For example, terpenes react with the ozone in the air we breath and create nasty toxins such as formaldehyde and the hydroxyl radical.  One set of researchers noted that "more than two dozen research articles present evidence of adverse health effects from inhalation exposure associated with cleaning or cleaning products." In other words, air fresheners have hurt people from even short term exposure, which makes long term exposure from lower levels a very real concern.
Furthermore, the cleaning products themselves have many documented toxins in them.  For example, glycol ethers can damage sperm,  2-Butoxyethanol induces liver tumors  and d-Limonene is toxic to kidneys.  Air fresheners also frequently use VOC's (Volatile Organic Compounds) such as benzene. Benzene is a compound in petroleum products that has been linked to cancer, especially leukemia.  One study found that Hispanics and African Americans were exposed to distubingly high levels of VOCs, one of which was 1,4-dicholobenzene from air fresheners. 
Of course, the list could go on and on, and scientists are still trying to get their arms around dosage and exposure levels. I would recommend tremendous caution in the meantime, because tens of millions of us were exposed to BPA and Phthalates for decades, while researchers were studying the ill effects. Is it really worth the risk just for the smell of lavender and vanilla in your living room?
WARNING: The hormonal life of your children could be at risk as well. See this NRDC Warning about how the phthalates spewing out of air fresheners could be putting your children, especially boys at risk. In my link on Bisphenol-A, I document some of the studies that have come out regarding the dangers of phthalates as well.
WARNING 2: Women are also likely damaging themselves when they buy air fresheners and cleaning products according to the results of one study which found double the breast cancer risk from use of such products. 
1) "Atmospheric Environment, Jun 2004, 38(18):2841-2865, "Cleaning products and air fresheners: exposure to primary and secondary air pollutants"
2) Indoor Air, Feb 7 2006, 16(3):179-191, "Cleaning products and air fresheners: emissions and resulting concentrations of glycol ethers and terpenoids"
3) Toxicology, Jun 1983, 27(2):91-102, "Reproductive toxicity of the glycol ethers"
4) Toxicological Sciences, 2002, 70:252-260, "Hepatic Effects of 2-Butoxyethanol in Rodents"
5) Regulatory Toxicology and Pharmacology, Feb 1991, 13(1):70-86, "The human relevance of the renal tumor-inducing potential of d-limonene in male rats: Implications for risk assessment"
6) American Journal of Industrial Medicine, Jan 2007, 7(5-6):403-412, "Projections of leukemia risk associated with occupational exposure to benzene"
7) Atmospheric Environment, Jun 2009, 43(18):2884-2892, "Ethnicity, housing and personal factors as determinants of VOC exposures"
8) Environmental Health, 2010, 9:40, "Self-reported chemicals exposure, beliefs about disease causation, and risk of breast cancer in the Cape Cod Breast Cancer and Environmental Study: a case-control study"