A Commute in Data: The comma2k19 Dataset

通勤数据：Comma2k19 数据集

https://arxiv.org/pdf/1812.05752v1
Abstract— comma.ai presents comma2k19, a dataset of over 33 hours of commute in California’s 280 highway. This means 2019 segments, 1 minute long each, on a 20km section of highway driving between California’s San Jose and San Francisco. The dataset was collected using comma EONs that have sensors similar to those of any modern smartphone including a road-facing camera, phone GPS, thermometers and a 9-axis IMU. Additionally, the EON captures raw GNSS measurements and all CAN data sent by the car with a comma grey panda. Laika, an open-source GNSS processing library, is also introduced here. Laika produces 40% more accurate positions than the GNSS module used to collect the raw data. This dataset includes pose (position + orientation) estimates in a global reference frame of the recording camera. These poses were computed with a tightly coupled INS/GNSS/Vision optimizer that relies on data processed by Laika. comma2k19 is ideal for development and validation of tightly coupled GNSS algorithms and mapping algorithms that work with commodity sensors.
摘要—comma.ai 介绍了 comma2k19，这是一个包含超过 33 小时加州 280 号高速公路通勤场景的数据集。该数据集包含 2019 个各为 1 分钟长的片段，记录了在加州圣何塞和旧金山之间 20 公里高速公路上的驾驶情况。数据集利用配备了与现代智能手机相似传感器的 comma EON 设备收集而成，这些传感器包括一个面向道路的摄像头、手机 GPS、温度计和一个九轴惯性测量单元（IMU）。此外，EON 设备还能捕获由汽车通过 comma grey panda 发送的原始 GNSS 测量数据和所有的 CAN 数据。文中还介绍了 Laika，这是一个开源的 GNSS 处理库，其提供的定位精度比收集原始数据时使用的 GNSS 模块高出 40%。该数据集包括了记录摄像头在全球参考框架中的姿态（位置和方向）估计，这些估计是通过一个紧密耦合的惯性导航系统/GNSS/视觉优化器计算得出的，该优化器依赖于 Laika 处理的数据。comma2k19 数据集非常适合用于开发和验证紧密耦合的 GNSS 算法和适用于普通传感器的地图算法。

I. INTRODUCTION

“Quality over quantity”, or that’s what they say anyway, but is this true in the world of data? The reality is that collecting data with high-end sensors is expensive as dedicated hardware is needed and this quickly becomes unfeasible for larger datasets. Affordable sensors on the other hand, are ubiquitous and already continuously logging data on billions of devices around the world. The world is a noisy place, some trends require big data to become obvious. To find such trends, algorithms need to be developed to deal with huge amounts of less than perfect data. It is this core idea that motivates comma.ai’s strategy to collect data with scalibility as a priority.
“质量胜于数量”，人们通常这么说，但在数据的世界里，这真的成立吗？实际情况是，使用高端传感器收集数据成本很高，因为需要专门的硬件，而且对于大规模数据集而言，这种方法很快就会变得不切实际。而价格可接受的传感器则随处可见，它们已经在全球数十亿设备上不断地记录着数据。世界充满了噪声，一些趋势只有通过大数据才能显现出来。为了发现这些趋势，需要开发能够处理大量不完美数据的算法。正是这个核心理念驱动了 comma.ai 优先考虑数据收集的可扩展性策略。
The dataset released here, comma2k191, contains data collected by an EON2 and a grey panda3 during 2019 minutes of driving sampled from a Californian commute (Figure 1). There are logs of a road-facing camera, a 9-axis IMU, the vehicle’s transmitted CAN messages and raw GNSS measurements. This makes this dataset uniquely valuable for the development of mapping algorithms that require dense data and can use raw GNSS data.
这里发布的数据集名为 comma2k19，它包含了在 2019 分钟的驾驶过程中，通过 EON2 设备和 grey panda 设备在加州通勤路线上收集的数据（见图 1）。数据记录包括一个面向道路的摄像头、一个九轴 IMU（惯性测量单元）、车辆传输的 CAN 总线消息，以及原始的 GNSS 测量数据。这些数据使得该数据集对于开发需要密集数据支持并且能够使用原始 GNSS 数据的地图算法来说非常宝贵。

GNSS数据怎么用啊？？？

Conventionally this is done by fusing global position fixes from a GNSS module with other sensors [1], [2], [3], [4]. However, these methods use a pre-computed navigation solution from the GNSS module, i.e. they are loosely coupled. A more optimal approach is to directly integrate the raw GNSS measurements into the mapping optimizer/filter, this is called tight coupling [5], [6], [7]. A tightly coupled GNSS/INS/Vision fusion algorithm is likely the state-of-theart global pose estimator for a commodity sensor package. The comma2k19 dataset is ideal to develop and validate such an algorithm.
传统上，这是通过将 GNSS 模块提供的全球定位修正与其它传感器数据结合起来实现的 [1]、[2]、[3]、[4]。但是，这些方法依赖于 GNSS 模块预先计算的导航解决方案，也就是说，它们之间的耦合是松散的。一个更优的方法是直接将原始的 GNSS 测量数据集成到映射优化器或滤波器中，这种方法称为紧耦合 [5]、[6]、[7]。紧耦合的 GNSS/INS/视觉融合算法可能是目前最先进的，用于商品化传感器套件的全局姿态估计技术。comma2k19 数据集是开发和验证此类算法的理想选择。
We also introduce Laika, an open-source GNSS processing library that was developed and validated using data from the comma2k19 dataset. Laika produces significantly more accurate position fixes than reported by the u-blox M8 GNSS module used for raw data collection.
我们还推出了 Laika，这是一个开源的 GNSS 处理库，它利用 comma2k19 数据集进行了开发和验证。与用于收集原始数据的 u-blox M8 GNSS 模块所报告的定位修正相比，Laika 能够提供显著更精确的位置修正。

II. RELATED DATASETS

There are several driving datasets in the literature such as KITTI [8], Cityscapes [9], RoboCar [10], ApolloScapes [11], Berkeley Deep Drive [12], including our previous public dataset [13]. Most of these datasets focus on high quality sensors such as LIDAR or high level computer vision annotations such as semantic segmentation, object detection and imitation learning.
文献中存在多种驾驶数据集，例如 KITTI [8]、Cityscapes [9]、RoboCar [10]、ApolloScapes [11]、Berkeley Deep Drive [12]，以及我们之前发布的公共数据集 [13]。这些数据集中的大多数集中在使用高质量传感器，例如激光雷达（LIDAR），以及高级的计算机视觉注释上，如语义分割、目标检测和模仿学习。
On the other hand, the dataset presented here focuses on consumer grade sensors for reproducibility and scalability. Additionally, all the data collected in this dataset is concentrated in a very small area, this high density ensures repeated observations of the same location across a variety of conditions. This combined with the raw GNSS logs makes this dataset more suited for the development of high performance localization and mapping algorithms intended to run on commodity hardware.
另一方面，这里介绍的数据集着眼于消费级传感器，以便复制和扩展。此外，该数据集收集的所有数据都密集在一个很小的地理区域内，这样的高密度确保了在不同条件下对相同地点的多次观测。这一点，加上原始的 GNSS 日志数据，使得该数据集更适合开发高性能的定位和地图算法，这些算法旨在普通硬件上运行。

III. SAMPLE CHOICE

The data was collected with the EON’s standard logging infrastructure. This specific highway was chosen because it is representative of the commute of millions of Americans that drive similar urban roads across the country every day. Data was only selected from this small portion of road to ensure that it is sufficiently dense for experiments mapping-related experiments. An interesting challenge of this dataset is that the vision data we collected is quite different from other datasets, in that there are a less good features to track[14] in the video. This makes it particularly interesting to test vision algorithms that need to work on the common highway driving scenarios.
数据是通过 EON 设备的标准日志记录功能收集的。选择这条特定的高速公路是因为它代表了数百万美国人每天驾驶的类似城市道路的情况。仅从这段道路上选取数据，以确保数据密度足以满足地图相关的实验需求。这个数据集的一个有趣挑战在于，我们收集的视觉数据与其他数据集相比有很大的不同，特别是在视频中可追踪的特征不多[14]。这使得它成为测试需要在普通高速公路驾驶场景中运行的视觉算法的绝佳材料。

IV. SENSOR SETUP

A. Vehicles

Data was logged on two different setups. A 2016 Honda Civic Touring and a 2017 Toyota RAV4 Platinum.
数据收集自两款车型。一款是 2016 年的 Honda Civic Touring，另一款是 2017 年的 Toyota RAV4 Platinum。

B. CAN messages

All the vehicles CAN messages are received and logged. Radar, steering angle and wheel speed readings have been parsed in this dataset.
所有车辆的 CAN 总线信息都被捕获并记录了下来。在这个数据集中，已经解析了雷达信号、转向角度和车轮速度的数据。

C. Camera data

The road-facing camera data was logged with a Sony IMX2984 camera sensor. Video is captured at 20Hz and compressed with H.264.
面向道路的摄像头数据使用了索尼 IMX298 传感器进行记录。视频以每秒 20 帧的频率捕获，并采用 H.264 编码进行压缩。这种设置旨在平衡数据的详细程度和存储效率，以便于处理和分析。

D. Raw GNSS

The grey panda, contains a u-blox M8 chip5 connected to a Tallysman TW4721 antenna. Raw data and u-blox’s navigation fix are logged at 10Hz. The raw data includes the doppler shifts, pseudoranges and carrier phases on the L1 channel for GLONASS and GPS. On the Civic the antenna was mounted inside the car under the windshield, on the RAV4 the antenna was mounted on the roof, resulting in a signal about 15dB stronger.
grey panda 设备内含一枚 u-blox M8 芯片，并连接到一枚 Tallysman TW4721 天线。原始数据和 u-blox 的导航修正以 10Hz 的频率被记录。这些原始数据包括了 L1 频道上 GLONASS 和 GPS 的多普勒频移、伪距和载波相位。在本田思域上，天线被安装在车内的挡风玻璃下，而在丰田 RAV4 上，天线被安装在车顶上，这导致信号强度大约增强了 15dB。

E. Other Sensors

Gyro and accelerometer data was collected with a LSM6DS3 at 100Hz and magnetometer data with a AK09911 at 10Hz. The EON also has an integrated WGR7640 GNSS receiver that also logs raw GNSS measurements in the same format as the u-blox module and logs at 1Hz. However, at least partly due to the bad antenna, the quality of the WGR7640 data is much lower.
陀螺仪和加速度计数据使用 LSM6DS3 传感器以每秒 100 次的频率收集，而磁力计数据则使用 AK09911 传感器以每秒 10 次的频率收集。EON 设备还集成了一个 WGR7640 GNSS 接收器，它也以与 u-blox 模块相同的格式记录原始的 GNSS 测量数据，记录频率为每秒 1 次。然而，至少部分原因是由于天线性能不佳，WGR7640 接收到的数据质量明显较差。

V. LAIKA

Laika6 is an open source GNSS processing library developed with comma2k19. Laika is similar to projects like [15] and [16], with a strong focus on simplicity, readability and straight-forward integration with other optimizers. Laika can be used to compute location fixes from the raw GNSS data that can be significantly more accurate than the live fix computed by GNSS module used for data collection.
Laika6 是一个开源的 GNSS 处理库，它是配合 comma2k19 数据集开发的。Laika 与 [15] 和 [16] 等项目类似，特别注重简洁性、易读性和与其他优化工具的直接集成。Laika 能够利用原始 GNSS 数据计算位置修正，这些修正的准确性可能显著高于数据收集时使用的 GNSS 模块所实时计算的修正。
To compute the fixes, raw measurements from the dataset are processed with Laika and then fed into a Kalman filter or an other optimizer that estimates positions. To prove the efficacy of Laika we used a simple Kalman filter that only takes GNSS data as input. A lack of ground truth can make it difficult to judge GNSS algorithms, since the true position of the receiver is never known. However, assuming the height of the road is constant within a small area, we can estimate the altitude accuracy of a position fix by checking the variation of estimated road height over small sections (5m x 5m) of road. This requires many passes through the same section of road to be reliable; luckily the high density data from this dataset is more than sufficient. Figure 3 shows the altitude error distribution for positions computed with Laika and the positions reported by the u-blox module. Overall the positioning error was reduced by 40%.
为了计算位置修正，首先使用 Laika 对数据集中的原始测量数据进行处理，然后将处理后的数据输入到卡尔曼滤波器或其他优化器中，以估算位置。为了验证 Laika 的效果，我们采用了一个只将 GNSS 数据作为输入的简单卡尔曼滤波器。由于接收器的确切位置从未被知晓，缺乏地面真实数据可能使得评估 GNSS 算法的有效性变得具有挑战性。但是，如果我们假设在一个小区域内道路的高度是恒定的，我们可以通过检查小段道路（5 米 x 5 米）上估计的道路高度变化来估算位置修正的高程精度。为了确保可靠性，这需要多次通过道路的相同部分；幸运的是，数据集中的高密度数据完全足够。图 3 展示了使用 Laika 计算出的位置与 u-blox 模块报告的位置的高程误差分布。总体来看，位置误差减少了 40%。


图 3 展示了 Laika 和实时 u-blox 基线算法在两种不同场景下的高程误差分布：一种场景是天线安装在车顶（左侧图），另一种场景是天线安装在车内（右侧图）。这些分布图用于比较 Laika 处理数据的方法与 u-blox 模块实时计算的精度，进而评估 Laika 在不同安装条件下的性能表现。

VI. GLOBAL POSES

In addition to the raw sensor data, the logs also contain best estimates for global pose (position + orientation) calculated by Mesh3D, comma.ai’s internal post-processing infrastructure that relies on data processed by Laika. They were computed with a tightly coupled GNSS/INS/Vision optimizer, where raw GNSS measurements and ORB [17] features were fed into a Multi-State Constraint Kalman Filter (MSCKF) [18], [19]. Figure 4 shows a snapshot of the resulting 3D path and lane estimates projected into camera frame.
除了原始的传感器数据，日志还包含了通过 Mesh3D 计算出的全局姿态（位置加方向）的最佳估计值，Mesh3D 是 comma.ai 的内部后处理系统，它依赖于 Laika 处理的数据。这些估计值是通过一个紧密耦合的 GNSS/INS/视觉优化器得出的，其中原始的 GNSS 测量数据和 ORB 特征被输入到一个多状态约束卡尔曼滤波器（MSCKF）中。图 4 展示了将计算出的 3D 路径和车道估计投影到摄像头坐标系的截图。

怎么能将规划的路径投影到图像数据呢？？？

The global position in the comma2k19 is given in ECEF [20] frame in meters, and the orientation is given as the quaternion that is needed to rotate from ECEF frame into local frame. Where the local frame is defined as [forward; right; down] in accordance with NED (North East Down) [21] conventions.
在 comma2k19 数据集中，全局位置以地心地球固定（Earth-Centered, Earth-Fixed，ECEF）坐标系的形式表示，单位是米，而姿态则通过四元数来定义，该四元数用于将位置从 ECEF 坐标系旋转到本地坐标系。本地坐标系是依据北、东、下（North East Down，NED）的约定定义的，其方向为“向前、向右、向下”。
To estimate the Root Mean Squared Error (RMSE) of the vertical component of position, we used the same technique as in Section V. By using the observed DOP[22] of each fix we can get a reliable estimate of horizontal errors too. To estimate the accuracy of the provided orientation, we took the Jacobian, J∆θi = @Ri=@∆θi, of the re-projection error ® for the ith observed ORB feature, with respect to orientation errors, ∆θ. We can then create linear equations to estimate the orientation error by using the Jacobian to linearize around ∆θi = 0. The high level equations used to calculate the RMSE of the orientation, θ^, are shown below (1).
为了估计位置垂直分量的均方根误差（RMSE），我们采用了与第五节相同的技术。通过使用每个位置修正点的观测到的几何稀释因子（DOP）[22]，我们同样可以得到水平误差的可靠估计。为了评估所提供姿态的准确性，我们计算了第 i 个观测到的 ORB 特征的重投影误差（R）相对于姿态误差 ∆θ 的雅可比矩阵 。然后，我们可以利用雅可比矩阵在 ∆θi = 0 处进行线性化，创建线性方程来估计姿态误差。用于计算姿态 RMSE，$\theta$ 的高级方程如下所示（1）。

Since most of the measured reprojection error, R, is due to noise in the ORB feature detection, it is fair to assume that (1) is an upper bound of the true orientation errors in our estimates. In Table I we show both estimated position and orientation errors.
由于大部分测量到的重投影误差 R 是由 ORB 特征检测中的噪声造成的，因此可以合理地假设方程（1）表示的是我们估计中真实姿态误差的一个上限。在表 I 中，我们列出了估计的位置误差和姿态误差。

Some applications require even more accurate poses than provided above. One can use vision to fine tune the pose estimates with a simple Expectation-Maximization algorithm: first average the ECEF position of the matching ORB features across image/pose pairs from different drives, this reduces the error in ORB feature localization. After that, we infer the corrected poses by relocalizing the frames against the averaged ORB features. An example of a single iteration of this type of correction is show in Figure 5.
有些应用场景需要比前文所述提供的姿态更为精确。可以通过使用简单的期望最大化（Expectation-Maximization）算法结合视觉数据来微调姿态估计：首先，对不同驾驶过程中匹配的 ORB 特征的 ECEF 位置进行平均，这有助于降低 ORB 特征定位的误差。然后，我们可以通过将帧与平均后的 ORB 特征重新进行匹配来推断出校正后的姿态。图 5 展示了这种校正方法单次迭代的示例。

VII. CONCLUSION

We proposed the comma2k19, a state-of-the-art dataset to develop and validate tightly coupled GNSS algorithms, fused pose estimators and mapping algorithms that are intended to work with commodity sensors. Using comma2k19 we built and open sourced Laika, a raw GNSS processing library that reduced positioning errors by 40% compared to the baseline algorithm shipped with the u-blox sensor used data collection. comma2k19 also includes camera poses in a global reference frame of the over 2 million images provided. We believe the most interesting future research directions using comma2k19 and Laika should be developing novel vision and sensor fusion based mapping algorithms for HD maps in highways with sparse features to track.
我们提出了 comma2k19，这是一个尖端的数据集，旨在开发和验证紧密耦合的 GNSS 算法、融合姿态估计器以及地图算法，它们都设计为与商用传感器配合使用。利用 comma2k19，我们构建并开源了 Laika，这是一个原始 GNSS 处理库，它将定位误差比 u-blox 传感器附带的基线算法降低了 40%。comma2k19 还包含了超过 200 万张图像的全局参考坐标系下的相机姿态。我们认为，使用 comma2k19 和 Laika 的最引人注目的未来研究方向应该是开发新型的视觉和传感器融合地图算法，这些算法适用于高速公路上特征稀疏的高精地图。

ACKNOWLEDGEMENT

We’d like to thank Eddie Samuels, Nicholas McCoy, George Hotz, Greg Hogan, Viviane Ford and Willem Melching for setting up the hardware and infrastructure that enabled this research.
我们要感谢 Eddie Samuels、Nicholas McCoy、George Hotz、Greg Hogan、Viviane Ford 和 Willem Melching，他们为设置硬件和基础设施做出了贡献，这些硬件和基础设施为这项研究提供了必要的支持。