improving tools readme (#20305)

* rough draft

* plotjuggler and carla description

* testing someting

* this is cool

* very short replay and debug desc

* small change

* small change 2

* better word

* low quality mode

* better word

* fixes

* changes

* minor fixes

* ubuntu_setup fix + README changes

* Update tools/README.md

* Update README.md

* Update README.md

* Update README.md

* changes

* Update tools/sim/README.md

* final changes

* final change

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: c4031750f3

tools/README.md

@@ -1,139 +1,71 @@
 openpilot tools
 ============
 
-tools to facilitate development and debugging of openpilot
+System requirements
-
-![Imgur](https://i.imgur.com/IdfBgwK.jpg)
-
-
-Table of Contents
-============
-
-<!--ts-->
- * [Requirements](#requirements)
- * [Setup](#setup)
- * [Tool examples](#tool-examples)
-   * [Replay driving data](#replay-driving-data)
-   * [Debug car controls](#debug-car-controls)
-   * [Stream replayed CAN messages to EON](#stream-replayed-can-messages-to-eon)
- * [Welcomed contributions](#welcomed-contributions)
-<!--te-->
-
-
-Requirements
 ============
 
-openpilot tools and the following setup steps are developed and tested on Ubuntu 20.04, MacOS 10.14.2 and, Python 3.8.2.
+openpilot is developed and tested on **Ubuntu 20.04**, which is the primary development target aside from the [supported embdedded hardware](https://github.com/commaai/openpilot#supported-hardware). We also have a CI test to verify that openpilot builds on macOS, but the tools are untested. For the best experience, stick to Ubuntu 20.04, otherwise openpilot and the tools should work with minimal to no modifications on macOS and other Linux systems.
 
 Setup
 ============
-1. Run `ubuntu_setup.sh` or `mac_setup.sh`, and make sure that everything completed correctly
+1. Clone openpilot into home directory:
-
-2. Compile openpilot by running ```scons``` in the openpilot directory
-   or alternatively run ```./openpilot_build.sh``` (uses a pre-configured docker container)
-
-3. Try out some tools!
-
-
-Tool examples
-============
-
-
-Replay driving data
--------------
-
-**Hardware needed**: none
-
-`unlogger.py` replays data collected with [dashcam](https://github.com/commaai/openpilot/tree/dashcam) or [openpilot](https://github.com/commaai/openpilot).
-
-Unlogger with remote data:
-
 ```
-# Log in via browser
+cd ~
-python lib/auth.py
 
-# Start unlogger
+git clone --recurse-submodule https://github.com/commaai/openpilot.git
-python replay/unlogger.py <route-name>
-#Example:
-#python replay/unlogger.py '3533c53bb29502d1|2019-12-10--01-13-27'
 
-# In another terminal you can run a debug visualizer:
-python replay/ui.py   # Define the environmental variable HORIZONTAL is the ui layout is too tall
 ```
 
-Unlogger with local data downloaded from device or https://my.comma.ai:
+2. Run setup script:
 
+Ubuntu:
 ```
-python replay/unlogger.py <route-name> <path-to-data-directory>
+openpilot/tools/ubuntu_setup.sh
-
-#Example:
-
-#python replay/unlogger.py '99c94dc769b5d96e|2018-11-14--13-31-42' /home/batman/unlogger_data
-
-#Within /home/batman/unlogger_data:
-#  99c94dc769b5d96e|2018-11-14--13-31-42--0--fcamera.hevc
-#  99c94dc769b5d96e|2018-11-14--13-31-42--0--rlog.bz2
-#  ...
 ```
-![Imgur](https://i.imgur.com/Yppe0h2.png)
+MacOS:
-
+```
-LogReader with remote data
+openpilot/tools/mac_setup.sh
-
-```python
-from tools.lib.logreader import LogReader
-from tools.lib.route import Route
-route = Route('3533c53bb29502d1|2019-12-10--01-13-27')
-log_paths = route.log_paths()
-events_seg0 = list(LogReader(log_paths[0]))
-print(len(events_seg0), 'events logged in first segment')
 ```
 
-Debug car controls
+3. Compile openpilot by running SCons in openpilot directory
--------------
+```
+cd openpilot && scons -j$(nproc)
+```
 
-**Hardware needed**: [panda](panda.comma.ai), [giraffe](https://comma.ai/shop/products/giraffe/), joystick
+4. Try out some tools!
 
-Use the panda's OBD-II port to connect with your car and a usb cable to connect the panda to your pc.
-Also, connect a joystick to your pc.
 
-`joystickd.py` runs a deamon that reads inputs from a joystick and publishes them over zmq.
+Tools
-`boardd` sends the CAN messages from your pc to the panda.
+============
-`debug_controls` is a mocked version of `controlsd.py` and uses input from a joystick to send controls to your car.
 
-Make sure the conditions are met in the panda to allow controls (e.g. cruise control engaged). You can also make a modification to the panda code to always allow controls.
+[Plot logs](plotjuggler)
+-------------
 
-Usage:
+Easily plot openpilot logs with [PlotJuggler](https://github.com/facontidavide/PlotJuggler), an open source tool for visualizing time series data.
-```
-python carcontrols/joystickd.py
 
-# In another terminal:
-BASEDIR=$(pwd) selfdrive/boardd/boardd
 
-# In another terminal:
+[Run openpilot in a simulator](sim)
-python carcontrols/debug_controls.py
+-------------
 
-```
+Test openpilots performance in a simulated environment. The [CARLA simulator](https://github.com/carla-simulator/carla) allows you to set a variety of features like:
-![Imgur](steer.gif)
+* Weather
+* Environment physics
+* Cars
+* Traffic and pedestrians
 
 
-Stream replayed CAN messages to EON
+[Replay a drive](replay)
 -------------
 
-**Hardware needed**: 2 x [panda](panda.comma.ai), [debug board](https://comma.ai/shop/products/panda-debug-board/), [EON](https://comma.ai/shop/products/eon-gold-dashcam-devkit/).
+Review video and log data from routes and stream CAN messages to your device.
 
-It is possible to replay CAN messages as they were recorded and forward them to a EON. 
-Connect 2 pandas to the debug board. A panda connects to the PC, the other panda connects to the EON.
 
-Usage:
+[Debug car controls](carcontrols)
-```
+-------------
-# With MOCK=1 boardd will read logged can messages from a replay and send them to the panda.
+
-MOCK=1 selfdrive/boardd/tests/boardd_old.py
+Use a joystick to control your car.
 
-# In another terminal:
-python replay/unlogger.py <route-name> <path-to-data-directory>
 
-```
-![Imgur](https://i.imgur.com/AcurZk8.jpg)
 
 Welcomed contributions
 =============
@@ -142,3 +74,5 @@ Welcomed contributions
 * Support for other platforms other than Ubuntu 20.04.
 * Performance improvements: the tools have been developed on high-performance workstations (12+ logical cores with 32+ GB of RAM), so they are not optimized for running efficiently. For example, `ui.py` might not be able to run real-time on most PCs.
 * More tools: anything that you think might be helpful to others.
+
+![Imgur](https://i.imgur.com/IdfBgwK.jpg)

tools/carcontrols/README.md

@@ -0,0 +1,26 @@
+Debug car controls
+-------------
+
+**Hardware needed**: [panda](panda.comma.ai), [giraffe](https://comma.ai/shop/products/giraffe/), joystick
+
+Use the panda's OBD-II port to connect with your car and a usb cable to connect the panda to your pc.
+Also, connect a joystick to your pc.
+
+`joystickd.py` runs a deamon that reads inputs from a joystick and publishes them over zmq.
+`boardd` sends the CAN messages from your pc to the panda.
+`debug_controls` is a mocked version of `controlsd.py` and uses input from a joystick to send controls to your car.
+
+Make sure the conditions are met in the panda to allow controls (e.g. cruise control engaged). You can also make a modification to the panda code to always allow controls.
+
+Usage:
+```
+python carcontrols/joystickd.py
+
+# In another terminal:
+BASEDIR=$(pwd) selfdrive/boardd/boardd
+
+# In another terminal:
+python carcontrols/debug_controls.py
+
+```
+![Imgur](steer.gif)

tools/replay/README.md

@@ -0,0 +1,67 @@
+Replay driving data
+-------------
+
+**Hardware needed**: none
+
+`unlogger.py` replays data collected with [dashcam](https://github.com/commaai/openpilot/tree/dashcam) or [openpilot](https://githucommaai/openpilot).
+
+Unlogger with remote data:
+
+```
+# Log in via browser
+python lib/auth.py
+
+# Start unlogger
+python replay/unlogger.py <route-name>
+#Example:
+#python replay/unlogger.py '3533c53bb29502d1|2019-12-10--01-13-27'
+
+# In another terminal you can run a debug visualizer:
+python replay/ui.py   # Define the environmental variable HORIZONTAL is the ui layout is too tall
+```
+
+Unlogger with local data downloaded from device or https://my.comma.ai:
+
+```
+python replay/unlogger.py <route-name> <path-to-data-directory>
+
+#Example:
+
+#python replay/unlogger.py '99c94dc769b5d96e|2018-11-14--13-31-42' /home/batman/unlogger_data
+
+#Within /home/batman/unlogger_data:
+#  99c94dc769b5d96e|2018-11-14--13-31-42--0--fcamera.hevc
+#  99c94dc769b5d96e|2018-11-14--13-31-42--0--rlog.bz2
+#  ...
+```
+![Imgur](https://i.imgur.com/Yppe0h2.png)
+
+LogReader with remote data
+
+```python
+from tools.lib.logreader import LogReader
+from tools.lib.route import Route
+route = Route('3533c53bb29502d1|2019-12-10--01-13-27')
+log_paths = route.log_paths()
+events_seg0 = list(LogReader(log_paths[0]))
+print(len(events_seg0), 'events logged in first segment')
+```
+
+Stream replayed CAN messages to EON
+-------------
+
+**Hardware needed**: 2 x [panda](panda.comma.ai), [debug board](https://comma.ai/shop/products/panda-debug-board/), [EON](https://comma.ai/shop/products/eon-gold-dashcam-devkit/).
+
+It is possible to replay CAN messages as they were recorded and forward them to a EON. 
+Connect 2 pandas to the debug board. A panda connects to the PC, the other panda connects to the EON.
+
+Usage:
+```
+# With MOCK=1 boardd will read logged can messages from a replay and send them to the panda.
+MOCK=1 selfdrive/boardd/tests/boardd_old.py
+
+# In another terminal:
+python replay/unlogger.py <route-name> <path-to-data-directory>
+
+```
+![Imgur](https://i.imgur.com/AcurZk8.jpg)

tools/sim/README.md

@@ -30,6 +30,14 @@ You can control openpilot driving in the simulation with the following keys
 To see the options for changing the environment, such as the town, spawn point or precipitation, you can run `./start_openpilot_docker.sh --help`.
 This will print the help output inside the docker container. You need to exit the docker container before running `./start_openpilot_docker.sh` again.
 
+## Performance
+
+openpilot doesn't have any extreme hardware requirements, however CARLA is very resource-intensive and may not run smoothly on your system. For this case, we have a low quality mode you can activate by running:
+```
+./start_openpilot_docker.sh --low_quality
+```
+You can also check out the [CARLA python documentation](https://carla.readthedocs.io/en/latest/python_api/) to find more parameters to tune that might increase performance on your system
+
 ## Further Reading
 The following resources contain more details and troubleshooting tips.
 * [CARLA on the openpilot wiki](https://github.com/commaai/openpilot/wiki/CARLA) 

tools/sim/bridge.py

@@ -17,10 +17,10 @@ from selfdrive.test.helpers import set_params_enabled
 
 parser = argparse.ArgumentParser(description='Bridge between CARLA and openpilot.')
 parser.add_argument('--joystick', action='store_true')
+parser.add_argument('--low_quality', action='store_true')
 parser.add_argument('--town', type=str, default='Town04_Opt')
 parser.add_argument('--spawn_point', dest='num_selected_spawn_point',
         type=int, default=16)
-parser.add_argument('--low_quality', type=bool, default=False)
 
 args = parser.parse_args()
 

tools/ubuntu_setup.sh

@@ -84,6 +84,7 @@ git submodule update
 pyenv install -s 3.8.5
 pyenv global 3.8.5
 pyenv rehash
+eval "$(pyenv init -)"
 
 # **** in python env ****