Filtering the default avatar list

I'm using the avatar_defaults filter hook to filter the default avatar list. For example:

function my_avatar_defaults( $avatar_defaults ) {
    $avatar_defaults['http://ift.tt/1zQvqaF'] = __( 'Foo' );
    return $avatar_defaults;
}
add_filter( 'avatar_defaults', 'my_avatar_defaults' );

This works because if I visit Settings > Discussion and then scroll down, I can see Foo has been added as a default avatar option. See the following screenshot for an example:

The problem is the src attribute of the image displayed next to Foo. It seems to be making a call to Gravatar. Here's an example of the source code I'm getting for the image next to Foo:

<img src="http://ift.tt/1zQvqqX" />

How can I ensure the src attribute points to my image URL instead of the Gravatar URL (which seems to have my image's URL inside it)?