870d19f33d
Reapply "File sourcing: Not all files are logs ( #36025 )"
...
This reverts commit 3570022b9a
2 months ago
3570022b9a
Revert "File sourcing: Not all files are logs ( #36025 )"
...
This reverts commit 18b7ddef8f
2 months ago
18b7ddef8f
File sourcing: Not all files are logs ( #36025 )
...
* Not all files are logs
* more refactor
* linting ok
* fix tests
* import exception
* whoops forgot to git add
* fix
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com>
2 months ago
5ec9aee216
File sourcing: simplify return type ( #36028 )
...
* rm str | none pattern
* clean up
* more clean up
* stash
* Revert "stash"
This reverts commit 3e2472160c
2 months ago
6005b12f94
format logreader
2 months ago
8ec61991ee
LogReader sourcing: remove redundant file existence checks ( #35991 )
...
* speed up sourcing but avoiding checking for existence of collected files already from previous sources
* clean up
* been meaning to make them return dicts
* no longer true
* no longer true
* clean up
* more
* more
* revert
3 months ago
1eef956cad
LogReader sourcing: return dict ( #35994 )
...
* new return type
* fix test
* why not
3 months ago
aa91a02db8
LogReader sourcing: check comma API source before CI source ( #35992 )
...
sort
3 months ago
e0f51bdbb6
Reapply "LogReader: wrap events to cache which() ( #35882 )" ( #35909 )
...
* Reapply "LogReader: wrap events to cache which() (#35882 )"
This reverts commit ba2dced54c
3 months ago
52a4b52628
FileName clean up ( #35938 )
...
two spaces!
3 months ago
d1e0a60408
Filename refactor: no enum ( #35930 )
...
* conflict
* typing
* typing
* no value
* fix typing
* whitespace
* whitespace
* unused
* Reapply "Filename: minor refactor (#35927 )"
This reverts commit 8c7d53004f
3 months ago
8c7d53004f
Revert "Filename: minor refactor ( #35927 )"
...
This reverts commit 96313fa4c0
3 months ago
96313fa4c0
Filename: minor refactor ( #35927 )
...
* Filename
* rest of refactor
3 months ago
ba2dced54c
Revert "LogReader: wrap events to cache which() ( #35882 )"
...
This reverts commit 0ebee55050
3 months ago
0ebee55050
LogReader: wrap events to cache which() ( #35882 )
...
* speed up lr
* lazy caching
* clean up
* it fast
* stash
* stash
* chatgpt code is bad as usual
* clean up
* clean up
* clean up
* clean up
* clean up
* clean up
* match behavior
* cmt
3 months ago
103b4df18c
Log filenames: make tuples ( #35771 )
...
* make tuples
* spelling: this linter is hardcore
3 months ago
ffc8ea5a86
LogReader: use file name constants ( #35765 )
...
use common
3 months ago
48892e339d
LogReader: remove testing closet source ( #35763 )
...
rm
3 months ago
278781e5af
LogReader: extend FileName ( #35760 )
...
extend filename
3 months ago
4a5385a5b0
LogReader: support connect URLs ( #35757 )
...
* support connect
* clean up
3 months ago
618a25a612
LogReader: simplify sourcing logic and improve AUTO performance ( #35753 )
...
* sourcing supports any file type!
* stuff
* no camera for now
* i guess these are fine without
* rename
* get rid of these too!
* fix
* fix
* this is better
* start to clean up!
* better
* holy
holy
* clean up sources
* more robust
* working but needs some clean up
* clean up
* remove some trash
* nl
* auto_source can only return if it finds acceptable logs from sources
* double negative is confusing
* default
* list isn't hashable
* fix typing
* clean up
* speed up -- test_models got zst before bz2 in openpilotci, so do that (some segments have both bz2 and zst!)
* don't be a hero
* same behavior for now
3 months ago
b50b351b15
LogReader: remove commaCarSegments selector ( #35750 )
...
* a source should not be a readmode... and readmode should be logtype
* fix that
* fixup
3 months ago
862a816215
LogReader: add data endpoint option ( #35751 )
...
* add data endpoint option
* clean up
4 months ago
c553c1f872
LogReader: support passing list of sources ( #35749 )
...
* far too long
* this is a better experience
* no rename for now
4 months ago
27d882cc0a
Revert "logreader.py: concurrent file existence checks" ( #34942 )
...
Revert "logreader.py: concurrent file existence checks (#34875 )"
This reverts commit e11b239b2e
7 months ago
e652f435f9
max_lat_accel.py: use parallel processing ( #34936 )
...
* use run_across_segments
* disable tqdm
7 months ago
e11b239b2e
logreader.py: concurrent file existence checks ( #34875 )
...
* concurrent file existence checks
* upper bound on parallel queries
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com>
7 months ago
65d667e60e
Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are missing" ( #34919 )
...
Revert "logreader: avoid repeated prompts in auto_strategy when rlogs are mis…"
This reverts commit b8229ebca6
7 months ago
b8229ebca6
logreader: avoid repeated prompts in auto_strategy when rlogs are missing ( #34890 )
...
* avoid repeated prompts in auto_strategy when rlogs are missing
* shorter
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com>
7 months ago
071f8ba658
LogReader: add time_series helper
8 months ago
9ec54f59c6
loggerd: switch logging from raw file format to zstd compressed files ( #34549 )
...
* switch logging from raw file format to zstd compressed files
* more zst suffix
* compress bootlog
* remove class RawFile
* Optimize ZstdFileWriter by adding input caching
* use ZSTD_compressStream2
* cleanup
* LOG_COMPRESSION_LEVEL=10
* space
* add zst suffix to LOGS_SIZE_RATE
9 months ago
2af9f68147
LogReader: more specific exceptions ( #33914 )
...
* more specific logreader exceptions
* huh
* fix
1 year ago
1e92af9b3c
[bot] Update Python packages ( #33436 )
...
* Update Python packages
* bump
* specify any list of sources
ooh this is pretty nice
* update
* y
* fix
* fix that
* bump
---------
Co-authored-by: Vehicle Researcher <user@comma.ai>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 239cc69a20
1 year ago
9da055853f
LogReader clean ups ( #33446 )
...
* buggin me
* no caps
* remove at least one level of indirection
* no recursion, remove a bunch of junk
* default is now allfault ??
* back
* fix tests
old-commit-hash: 5796bf12b6
1 year ago
730924e7f2
LogReader: formatting and remove unnecessary type alias
...
old-commit-hash: 34305bec09
1 year ago
786aff27da
azure_container: pass the filename explicitely when using get_url ( #33407 )
...
* Pass filename explicitely when using azure container
* Do not update segments in test_processes
* openpilotci_source_zst
* Fix static analysisc
old-commit-hash: b5958ebb60
1 year ago
937ed5f700
Add testing closet to LogReader source ( #33242 )
...
* add testing closet to logreader source
* check source availability
* use internal_source_available
old-commit-hash: 118932acd3
1 year ago
6a2b4dedfc
tools/logreader: add desc arg to run_across_segments() ( #33226 )
...
old-commit-hash: 9e648e428d
1 year ago
7ce33fbe2b
process replay: use zst ( #33138 )
...
* test
* update refs to zst
* update
old-commit-hash: dfd387520e
1 year ago
7b67994a05
use zstandard library ( #33110 )
...
* use zstandard
* run 10
* bye bye
* this was a little overzealous
* clean up
old-commit-hash: 1b3b5ab758
1 year ago
539c828861
tools: keep re pattern file small ( #33084 )
...
remove bz2 from re file
old-commit-hash: e68bb26e14
1 year ago
309ccf6482
LogReader: support multiple log extensions in route ( #33066 )
...
* support rlog.zst
* TODO
old-commit-hash: e38a1428aa
1 year ago
5333e8ba20
LogReader: show total segment count in missing rlog prompt
...
old-commit-hash: 3e41b1b287
1 year ago
5887093eed
LogReader: print number of missing rlogs ( #32830 )
...
* count rlogs
* minimization changes
* cache to avoid duplicate checks
* no real point in checking qlogs to just show the warning
---------
Co-authored-by: Shane Smiskol <shane@smiskol.com>
old-commit-hash: 504dd4543d
1 year ago
9f50db15c0
LogReader: try zst on internal source ( #32751 )
...
* internal source list files like azure api
* messy but works
* no limit
* simpler
* clean up
* clean up
* clean up
* that's obvious
* better
* we need to unfortunately return a url, so best to take a naive approach for now
* todo
* fix
* clean up
old-commit-hash: b45caf4033
1 year ago
644156420b
LogReader: revert broken check
...
it returns files OR None
old-commit-hash: 68e22faa29
1 year ago
8f402f29a3
SegmentRange: remove timestamp ( #32750 )
...
* add deprecation warning
* remove timestamp
* Update logreader.py
old-commit-hash: d5ce4b3085
1 year ago
ed80e9357a
LogReader: improve error messages ( #32747 )
...
* better error messages
* clean up
old-commit-hash: 3ede1e2a7c
1 year ago
e88cbb0162
add zstd Python library ( #32731 )
...
* add zstd
* fix
* uploader
* logreader: zst support
* caps
* only logreader
old-commit-hash: 04a93dd849
1 year ago
450d012166
logreader.py: optimize event loading and sorting in _LogFileReader ( #32729 )
...
* improve logfilereader
* less changes
old-commit-hash: c64bca97ba
1 year ago
Shane Smiskol
Maxime Desroches
Harald Schäfer
Adeeb Shihadeh
Dean Lee
commaci-public
Kacper Rączy
Jimmy
Hoang Bui
StefanLekanic-SYRMIA