The tumor microenvironment (TME) comprises an assortment of immune and non-immune cells. The interactions between the cancer cells and their surrounding TME are known to be a cardinal factor in all stages of cancer progression, from initiation to metastasis. Tumor-associated macrophages (TAMs) and cancer-associated fibroblasts (CAFs) are considered two of the most abundant TME members associated with poor prognosis in various cancer types. Intercellular communication between the cancer cells and TME cells might occur via direct cell-cell contact or achieved through secreted factors such as cytokines, growth factors and extracellular vesicles (EVs). EVs are released by almost every cell type and by cancer cells in particular. EVs are loaded with unique molecular cargos that might include DNA, proteins, RNA and lipids, commonly reflecting the physiological traits of their donor cells. Once released, EVs are capable of initiating short- and long-distance communication in an autocrine, paracrine and endocrine fashion. The molecular cargos within the EVs are able to impart phenotypic changes at the receiving end thus allowing EV-releasing cancer cells to deliver messages to TME cells and tighten their grasp over the cancerous tissue. In this concise review, we aim to document the bidirectional EV-based communication between cancer cell, TAMs and CAFs, tilting the balance in favor of cancer progression and metastasis.