Permanent magnet array is a welcome option to provide main magnetic field for portable magnetic resonance imaging (MRI). In this abstract, we propose an efficient and fast optimization method which can optimize the filed strength and homogeneity for the design of permanent magnet arrays. The magnetic field of permanent magnets with the interference of irons is calculated by applying boundary integral method (BIM). For optimization, genetic algorithm particle swarm optimization (GAPSO) is applied which offers highly diversified options and converges fast. A permanent magnet array is optimized with significantly improved performance, and it will be built for low-field portable MR imaging.