Wordpress Custom Widget Strange Behaviour

I have developed a Custom widget plugin with a dozen of input fields. Some of those input fields load Fabtastic Color Picker. When I add the widget in any widget area, the color picker loads very slow. But after clicking on the SAVE button, the fields load without any delay ...

Is this a bug in wordpress widgets or in my plugin?