There are two parts to backing up your WordPress site: Database and Files.
Your WordPress site consists of the following files:
- WordPress Core Installation
- WordPress Plugins
- WordPress Themes
- Images and Files
- Additional Files and Static Web Pages
Everything that has anything to do with the look and feel of your site is in a file somewhere and needs to be backed up. Additionally, you must back up all of your files in your WordPress directory (including subdirectories) and your .htaccess file.
While most hosts back up the entire server, including your site, it is better that you back up your own files. The easiest method is to use an FTP program to download all of your WordPress files from your host to your local computer.
By default, the files in the directory called wp-content are your own user-generated content, such as edited themes, new plugins, and uploaded files. Pay particular attention to backing up this area, along with your wp-config.php, which contains your connection details. The remaining files are mostly the core files, which are supplied by the WordPress download zip file.
Normally, there would be no need to copy the WordPress core files, as you can replace them from a fresh download of the WordPress zip file. The important files to back up would be your wp-config.php file, which contains your settings and your wp-content directory (plus its contents) which contains all your theme and plugin files.
As with other backup plugins there are some real opportunities for overloading a server. In most cases backing up themes and plugins is only necessary when you add, replace, or upgrade them. Make it a policy to back them up at that time.
Files in your uploads folder may have much more regular activity than plugins and themes so backing them up regularly may be helpful.
A plugin I recommend for this is WordPress Backup by BTE
Update 2014: In the ever changing world of WordPress plugins, another favorite has been abandoned by its author. We now recommend: