Today, Firehose uploads and user-requested uploads from Connect are treated the same. This is not ideal behavior if the user wants to upload routes immediately for a bug report and the queue is full of uploads for Firehose. The workaround is to clear the queue and retry the upload from Connect.
This PR adds an optional `priority` to requested file uploads in `athenad`. By default, all requests are marked `Low`. However, if the caller wishes to mark their uploads as "more important," then the upload queue will prioritize those requests when uploading.
The only caveat to this PR is that we won't reorder files currently being uploaded. Most connections are quick enough to finish uploading max 4 files before polling the new high-priority items in the queue.
Closes https://github.com/commaai/openpilot/issues/34836
**Verification**
Added test case to insert upload tasks with differing priorities. Polling the queue produces items in the correct order (smallest # to largest).
PR to mark Connect uploads as high priority: https://github.com/commaai/connect/pull/557
---------
Co-authored-by: Cameron Clough <cameronjclough@gmail.com>
* zstd uploader
* fix that
* fix name of function
* comment
* log failed
* fix comma_api_source for routes with both bz2 and zst rlogs
* TODO
* 10-14 achieves almost no benefit on qlogs in a few cases, but takes 2x the time
* these aren't written out
* regen: specify any list of sources
ooh this is pretty nice
* regen and process replay
* damn, actually we don't need all this (cool tho)
Revert "regen: specify any list of sources"
This reverts commit ceb0b4abed.
* just let it auto resolve
* fix athenad/uploader tests
* zst here too
* TODOs
* yes
* Revert "TODOs"
This reverts commit 8c7da1dbd0.
* Revert "zst here too"
This reverts commit 23b0023ddf.
* Revert "just let it auto resolve"
This reverts commit f296d62424.
* Revert "regen and process replay"
This reverts commit 0768330e96.
* revert readme
* not in save_log either
* lfg
* Revert "lfg"
This reverts commit 3718559c6c.
old-commit-hash: 7dec7c39be
* useless
* Revert "useless"
This reverts commit 28f0bb9e97.
* this forever hangs you disconnect (or 2 hours)
* same timeout as the global websocket
* Revert "same timeout as the global websocket"
This reverts commit 0bd0cb8a38.
* setting the timeout affects the entire websocket and disconnects, not just recv timeout
* fix that
* fix test
old-commit-hash: 09aeab3f77
* check end_event while uploading, throw abort exception if we need to shut down/restart
* fix
* draft test
stash
* Revert - there's no easy way to know if it breaks early in upload loop
or not yet
This reverts commit ad893687e1.
* todo for now
old-commit-hash: 3009a51c06
* these pass in 0.5s since server sends ping on connect
* comments
* unused
* fix
* fix these too
* check end_event while uploading, throw abort exception if we need to shut down/restart
* Revert "check end_event while uploading, throw abort exception if we need to shut down/restart"
This reverts commit f0b822fca9.
* more tol for lte connection
old-commit-hash: c0e172e0c0
* parallel tests
* review suggesions
* add to pyproject
* add a bit more buffer on that
* fix rare athena issue
* remove from pypoetry
old-commit-hash: dde225221e
* use OP prefix for logmessage
* cleanup paths too
* cleanup the paths too
* add hw.py to release
* fix those issues
* fix unittests
* fix unittests
* fix unittests
* do swaglog_ipc properly across all the files
* fix that
* fix swaglog in c++
* review suggestions
old-commit-hash: bfe990b112
Trey Moen
Adeeb Shihadeh
BrainLess
Dean Lee
Shane Smiskol
Justin Newberry
Cameron Clough
Greg Hogan