Today, Firehose uploads and user-requested uploads from Connect are treated the same. This is not ideal behavior if the user wants to upload routes immediately for a bug report and the queue is full of uploads for Firehose. The workaround is to clear the queue and retry the upload from Connect.
This PR adds an optional `priority` to requested file uploads in `athenad`. By default, all requests are marked `Low`. However, if the caller wishes to mark their uploads as "more important," then the upload queue will prioritize those requests when uploading.
The only caveat to this PR is that we won't reorder files currently being uploaded. Most connections are quick enough to finish uploading max 4 files before polling the new high-priority items in the queue.
Closes https://github.com/commaai/openpilot/issues/34836
**Verification**
Added test case to insert upload tasks with differing priorities. Polling the queue produces items in the correct order (smallest # to largest).
PR to mark Connect uploads as high priority: https://github.com/commaai/connect/pull/557
---------
Co-authored-by: Cameron Clough <cameronjclough@gmail.com>
* switch logging from raw file format to zstd compressed files
* more zst suffix
* compress bootlog
* remove class RawFile
* Optimize ZstdFileWriter by adding input caching
* use ZSTD_compressStream2
* cleanup
* LOG_COMPRESSION_LEVEL=10
* space
* add zst suffix to LOGS_SIZE_RATE
* zstd uploader
* fix that
* fix name of function
* comment
* log failed
* fix comma_api_source for routes with both bz2 and zst rlogs
* TODO
* 10-14 achieves almost no benefit on qlogs in a few cases, but takes 2x the time
* these aren't written out
* regen: specify any list of sources
ooh this is pretty nice
* regen and process replay
* damn, actually we don't need all this (cool tho)
Revert "regen: specify any list of sources"
This reverts commit ceb0b4abed.
* just let it auto resolve
* fix athenad/uploader tests
* zst here too
* TODOs
* yes
* Revert "TODOs"
This reverts commit 8c7da1dbd0.
* Revert "zst here too"
This reverts commit 23b0023ddf.
* Revert "just let it auto resolve"
This reverts commit f296d62424.
* Revert "regen and process replay"
This reverts commit 0768330e96.
* revert readme
* not in save_log either
* lfg
* Revert "lfg"
This reverts commit 3718559c6c.
old-commit-hash: 7dec7c39be
* check end_event while uploading, throw abort exception if we need to shut down/restart
* fix
* draft test
stash
* Revert - there's no easy way to know if it breaks early in upload loop
or not yet
This reverts commit ad893687e1.
* todo for now
old-commit-hash: 3009a51c06
* enable random order
* logprint info
* revert that
* better logging
* go to dfu
* try to flash every panda if not up to date
* revert panda debugging
old-commit-hash: a5d99e0ebb
* use OP prefix for logmessage
* cleanup paths too
* cleanup the paths too
* add hw.py to release
* fix those issues
* fix unittests
* fix unittests
* fix unittests
* do swaglog_ipc properly across all the files
* fix that
* fix swaglog in c++
* review suggestions
old-commit-hash: bfe990b112
* switch to pytest
* static analysis
* fix the helpers
* static analysis
* remove that
* more parallel test
* cleanup
* static analysis
* durations min and tolerance
* reduce those iterations
* sleep time could be decreased too
* in case it's negative
* try using caching
* remove parallelization for first PR
* PR Cleanup
* reduce diff
* fix that
* bump panda
old-commit-hash: ca435d3447
* remove log_name
* log without compression
* fix tests
* remove extension for bootlog
* another test fix
* uploader compresses
* also compress in athena
* only compress qlog
* more generic check in do_upload
* fix bootlog compression
* lower loggerd cpu usage
* dont link against bz2
* set core affinity to little cluster
* handle old files
old-commit-hash: 77a6f3d034
* athena: uploads, show which items fail on error
* fix upload-id
* no more 404
* Update selfdrive/athena/athenad.py
Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 3ffebf4df5
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default
Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
old-commit-hash: 332f568a82
Trey Moen
Dean Lee
Shane Smiskol
Maxime Desroches
Adeeb Shihadeh
cl0cks4fe
Justin Newberry
Cameron Clough
Willem Melching
Maykon Pacheco
Ryan
Joost Wooning
Ewout ter Hoeven
Devin Leamy
Greg Hogan