X-ray binaries (XRBs) are systems containing accreting compact objects (black holes and neutron stars) that launch powerful relativistic jets. These binary systems evolve through bright outburst phases on rapid timescales of days to months, allowing us to probe jet and accretion phenomena in real time. Recently, it has been shown that we can move beyond spectral/imaging studies of these systems, and use time-domain observations to address the key open questions in compact object research. However, X-ray binary timing studies in the optical regime currently lag behind those performed at other wavelengths. We propose the GOFAST-XRB (Gemini Optical FAST timing of X-Ray Binaries) Large Program, which will use Gemini's `Alopeke and Zorro instruments to perform the first coherent census of XRB optical variability across different compact objects, timescales (sub-seconds-months), and accretion regimes. These Gemini data will be paired with NICER X-ray observations to track matter propagating from inflow to outflow in X-ray binaries, and gain insights into jet physics, particle acceleration, and how these processes relate to the accretion flow.