Since nobody else has mentioned it, I found that the xdebug debugger dramatically increased the time. I served a basic "Hello World, the time is 2020-01-01T01:01:01.010101" dynamic page and used this in my httpd.conf to time the request:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" **%T/%D**" combined
%T is the serve time in seconds, %D is the time in microseconds. With this in my php.ini:
[XDebug]
xdebug.remote_autostart = 1
xdebug.remote_enable = 1
I was getting around 770ms response times, but with both of those set to 0 to disable them, it jumped to 160ms instantly. Running both of these brought it down to 120ms:
php artisan route:cache
php artisan config:cache
The downside being that if I made config or route changes, I would need to re-cache them, which is annoying.
As a sidenote, oddly, moving the site from my SSD to a spinning HDD provided no performance benefits, which is super odd to me, but I suppose it's maybe cached, I'm on Windows 10 with XAMPP.