You usually have to change the width and height in BOTH, the format tab, and the Position Tab. If that still doesn't work, you can try running CamsSream, and set your resolution there first. Then, go back to WCS, and set your Position tab height and width before before bringing in your camera.
I have the same problem on WCS 0.74. I'm on linux Mint and trapped at the 320x200 pixel addressability (aka resolution).
Did you ever get the answer?
I read somewhere that the $HOME/.webcamstudio/ folder has config data, but I only found a script that contained a command with a height/width argument. I tried changing it to 800x600, then launched WCS, but it had no effect, and in fact WCS overwrote those files and it got set back to 320x200.