Let w be the width and h the height of your original tile. Next you have to decide the integer parameters a, b, m and n. These determine how many times your original image should be tiled to form the new image. Take a look at the picture below to get the idea what these parameters mean. Next calculate the s and D as shown (if D is negative then there is no solution for your parameters). Finally, calculate the beta (usually the minus sign gives better results) and alpha angles as well as x and y that are the width and height of the resulting image. Alpha gives the angle of shear and beta the angle to rotation (positive sense is the one show in the picture). These values can be directly used in Adobe Illustrator and in some other programs as well.