Sort the rest. Divide items between those you need to carry all the time (keys, sunglasses, cell phone, pen, notepad, etc.) and those that are questionable - although any one item may not take up much space or add much weight, collectively these items can have a big impact:  

Makeup: Maybe you can just carry your lipstick, and keep the rest of your touch-up items at work so you don't have to schlep them back and forth from home each day.

Loose change: Unless you're saving it for a serendipitous encounter with a slot machine, put that loose change at the bottom of your purse or bag into your wallet or pants pocket where you can use it, or into a jar at home if you want to save it.

Checkbook: If you only write checks occasionally, maybe you can carry just a check or two in your wallet. Use carbon checks to serve as a reminder to note the check in your check register.

Paperwork: Take the appropriate action to stop carrying these items around:

Bills: If you carry unpaid bills more than a day or two, maybe you should block out 15 minutes or so on your calendar and just get them paid. Once you pay them, file or toss them as you see fit - don't continue to carry them around. 

Coupons: Clean out expired coupons, and consider whether you need to carry the others with you - maybe you can keep store coupons in your car's glove compartment and grocery coupons at home with your shopping list. If you don't have a shopping list, we need to talk. 

Receipts: Clean out receipts once a week at a minimum. Unless you're about to return something and need the receipt, you don't need to have them with you. Establish a place where you can collect them and find them when you need them. 

Phone numbers: Transfer the phone numbers you've jotted onto scraps of paper into your address book, whether it's paper or electronic. If you don't have an address book, now is the time to start one.

Lottery tickets: There are three possibilities: the drawing hasn't been held yet, your ticket is a winner, or it's not.  Act appropriately and move those tickets along on their journey.

Organize the contents. Rather than just tossing everything back in your purse or bag, try to compartmentalize or containerize it. This will make it easier to find things, and also make it easier to move items from one purse to another. Take advantage of the zippered pockets or dividers your purse or bag may have to stash smaller items.  Otherwise, use zippered pouches or even plastic bags for those smaller items. 

Clean it out regularly. Empty out loose change, receipts and random bits of paper every day, if possible, or once a week at a minimum.