If the coolant in a refrigerator was not compressed back into a liquid after it flowed through the refrigerator what would happen

The system would stop cooling, and simply recycle warm vapor through the refrigeration loop.

Explanation:

From the Mollier diagram we see that one part of the "refrigeration" loop is not complete. The step from 1 to 2 in the diagram would not occur.

This may occur due to a lack of sufficient refrigerant in the system or a faulty compressor. In any case, this results in warm "coolant" vapor passing into the receiver without any condensation (Step 3-4 cannot happen).

Without any liquid coolant available, the cooling effect of expansion of the liquid through the evaporator is lost (step from 5-6), and the system will simply use energy (and generate more heat) while recirculating warm vapor.