Cell competition, a fitness sensing process is essential for tissue homeostasis. Employing cancer metastatic latency models, we show that cell competition results in displacement of latent metastatic (Lat-M) cells from the primary tumor. Lat-M cells resist anoikis and survive as residual metastatic disease. Remodelled extracellular matrix facilitates Lat-M cell displacement and survival in circulation. Disrupting cell competition dynamics by depleting SPARC reduced displacement from orthotopic tumors and attenuated metastases. In contrast, depletion of SPARC post-extravasation in lung resident Lat-M cells increased metastatic outgrowth. Furthermore, multi-regional transcriptomic analyses of matched primary tumors and metachronous metastases from kidney cancer patients identified tumor subclones with latent metastatic traits. Kidney cancer enriched for these latent metastatic traits had rapid onset of metachronous metastases and significantly reduced disease-free survival. Thus, an unexpected consequence of cell competition is displacement of cells with latent metastatic potential, thereby shaping metastatic latency and relapse.