A generic error occurred in GDI+

This is a common exception you'll get when trying to delete or rename an image file. The problem is that the method to load the image from file will put a lock on the file, so you can't do anything with it until you dispose the image.
There are two ways around that: either load the image from a stream or dispose the image before deleting or renaming the file.
Loading the image from stream works, but the actual loading is a bit slower and you need to keep the stream open until you don't need the image anymore. You can't just load it with a stream, close it, and then work with the image. It might work with some file format, but the documentation itself tells you to keep the stream open. Here's how to do it:

using (FileStream stream = new FileStream(@"path\to\image", FileMode.Open, FileAccess.Read))
{
    pictureBox1.Image = Image.FromStream(stream);
}

The problem here is that after assigning the image to the picturebox the stream will close ("using" statement). You could do this instead:

FileStream stream = new FileStream(@"path\to\image", FileMode.Open, FileAccess.Read);
pictureBox1.Image = Image.FromStream(stream);

But then you'll have an open stream forever, until you close it, which is not really a good solution. The real solution comes from loading the image from file, like this:

pictureBox1.Image = Image.FromFile(@"path\to\image");

And then dispose it correctly.
I'm sure you know that we should always dispose everything disposable after we're done with it, so it shouldn't be a surprise, but the problem in disposing and then deleting or renaming soon after is that the image has not enough time to be disposed. So, if you do this:

pictureBox1.Image.Dipose();
File.Delete(@"path\to\image");

You'll most likely incur in the "A generic error occurred in GDI+" exception, because the lock on the file is not yet released.
The REAL and FINAL solution to this, is to call the garbage collector before deleting the file.
You can do that by writing this:

GC.Collect();

After disposing the image.
There's still a problem though, the Collect() method is called asynchronously, that means you could and probably will incur in the same error as before, since the file could still be locked. You need to tell the program to wait for the garbage collector to finish, by adding this:

GC.WaitForPendingFinalizers();

After the GC.Collect().
So, the final code would be:

pictureBox1.Image.Dipose();
GC.Collect();
GC.WaitForPendingFinalizers();
File.Delete(@"path\to\image");

And that's how you delete or rename an image file safely in C#.