Questions: 1. How hard is it to devise realistic transformations in domains beyond image recognition? Examples? We have also tried realistic transformations on malware samples such as PDF or Android app. We first examine the features of the ML/DNN classifier used for classifying input as benign or malicious, then we constrain the gradient descent to only allow changes to the features that won't affect the malicious functionality. For example, we allow adding features to Android manifest file of some permissions, etc. We think this transformation is realistic as it can be easily achieved by the attacker without changing maliciousness of the input, but hopefully, DeepXplore can find these generated inputs that induce some DNNs to think it is benign. 2. Neuron coverage seems coarse grained, like line-based code coverage metrics. Could path-based coverage (or even just edge-based coverage) be useful? Yes, neuron coverage defined here is more like a statement coverage. Path-based coverage may make more sense as covering a set/path of neurons represent a particular class of inputs. Then generating inputs that cover different paths of neurons may help to generate more "diverse input" that hopefully reveal more corner-case errors/bugs. 3. How is the transformation (like lightening the image) selected? Lightning can be easily achieved by modifying the original gradient descent method by a little bit (projected gradient descent). In fact, we found DeepXplore can actually support a much wider range of realistic transformation on the image than what is included in the original paper, such as rotation, translation, reflection, etc. Feel free to contact me for more details 4. Can you suggest where/what is wrong in the neural network? As decision logic of neural network is too complex and currently cannot be completely comprehensible by humans. So currently we can only reveal the potential errors, and one possible way to fix them is to feed these errors back to the training set and ask the neural network to retrain. A good analogy is currently neural network is like a child brain, when a child makes mistakes, we have no way to open his/her brain and tweak the neurons and guarantee he/she won't make same mistakes. The only way we can do now is to tell the kid this is a mistake by a teacher, then hopefully next time he/she won't make the same mistakes, but without guarantees. There is an active area and we are also currently working towards giving formal guarantees on correctness and improving the robustness of DNNs. 5. Self-driving ML systems typically rely on multiple sensors, not just images. Can your tool help with multiple input sensors? Yes, as long as the inputs are fed to the neural network for perception and decision-making, our tool can help to find inputs that may be corner-case errors. 6. Why can the same DNN algorithm using the same dataset generate different decisions? Implementation bugs? First, there are usually a large number of different DNNs with different architectures trained on different datasets. to deal with same tasks, e.g., different companies train their own models for driving cars and labeling images. Even same DNN algorithm using the same dataset still depends on many random factors (hyperparameters) such as the random initialization of the weight parameters in the neural network, the number of batch size, the number of training epochs, which always lead to slightly different trained DNNs. 7. Input generation and mutation play a key important role in finding the differential outputs from different DNNs. How this can be generalized? Just like test case generation, we can generate inputs without labels that reveal corner-case errors. As we don't need labels, then input can be taken by any sensors in the real world and can be huge. This idea of differential testing is generalizable to many other applications such as machine translation, where manual labeling effort is expensive. 8. How did you check the all nodes activated or not? According to the paper, you used the feature of Tensorflow, can you specifically explain about it? You can actually define a function using Tensorflow/Keras that takes the input of DNN as input, and output of a specific neuron as output. Then you simply feed the original input to this newly defined function and get the output of this function, which is exactly the output of an intermediate neuron's output in the original DNN. 9. Most of your corner cases were based on images washing out details from the input. Can we say too much variation from the original input makes it different? This depends on the definition of "too much". Yes, if you define a simple Euclidean distance between pixels of original and transformed image, this change may be too much. But if defining with respect to human perception, lighting effect, occlusion or other image distortion or deformation bounded by some factor should not introduce decision changes. We are looking for the transformations that remain invariant in human or other oracle but make DNNs make different predictions. Indeed, this requires some well-defined domain knowledge before performing testing. 10. A vision sentence: most improvement come from bugs. Decision making is right or wrong, how to exactly define? Currently, we define the behavior discrepancy as multiple DNNs differ in their most probable output (e.g., output label with the highest confidence). Indeed, it is not always true. For example, DNNs sometimes output multiple labels (e.g., top-5) all with similar high confidence, as the input includes multiple classes, e.g., having both cat and dog. Then maybe one DNN outputs slightly higher confidence saying it is a cat, but another says it is a dog with a little higher confidence. In this case, we should not think any of these DNNs make an obvious mistake because the input does contain both cat and dog. I agree that the "decision making is right or wrong" can be defined in a more rigorous way given the complexity of the input. 11. Seems this is like fuzzing, right? And how did the system mutate the seed? Randomly or under some guidelines? This exactly fuzzing in a more guided way (like a symbolic execution on DNNs). Traditional fuzzing with whitebox access can be guided by coverage but still using some heuristics (genetic algorithm). We leverage the differentiable property of DNN such that we can use gradient descent to find the input for both covering different neurons and generating corner-case errors/discrepancies.