WP_User_Query orderby meta_val_num

I know that normal WP_Query's can have an "orderby" parameter of "meta_value_num", but when I try to user "orderby" => "meta_value_num" on a WP_User_Query, it doesn't seem to work. Anyway I can obtain this functionality or is there something I'm missing?