Living in a home means having bills. Electric bills, gas bills, phone bills, water bills, and all manner of other services that come to mind. Then there's insurance and bank statements and such. And that usually means paper... invited or not, there is a way to make it easier.
For regular utilities, why not sign up for paperless bills online? You can even get bank statements online. You can keep notices and important papers filed right in a web based email account (so you can access it from any computer with internet) and never worry about having to file beyond that. When and if needed, go and print what you need if you need proof of payment.
Having less paper come into my house means less work for me. Sure I have paper files for the few things that do come in for utilities for example. It's a very thin file and if I wanted to eliminate that, I could even scan and "file" those things on a computer...and still be able to print those out, if it ever becomes necessary.
Check out MyCheckFree.com for a easy way to pay regular bills. They even have a way for you to schedule your payment based on the best time for you (before the due date, of course). For other bills not listed there, ask the companies that you do business with to see if you can go paperless with them. You save a stamp each time or even save gas if you are used to paying in person.
For questions about privacy, check with each company you deal with for details. Make sure that the sites are protected when it comes to your details and personal information.