In Cao, Du, Li and Li [9], a nonlocal diffusion model with free boundaries extending the local diffusion model of Du and Lin [18] was introduced and studied. For Fisher-KPP type nonlinearities, its long-time dynamical behaviour is shown to follow a spreading-vanishing dichotomy. However, when spreading happens, the question of spreading speed was left open in [9]. In this paper we obtain a rather complete answer to this question. We find a threshold condition on the kernel function such that spreading grows linearly in time exactly when this condition holds, which is achieved by completely solving the associated semi-wave problem that determines this linear speed; when the kernel function violates this condition, we show that accelerated spreading happens.