We recently discovered that the IS2RE results using the Relaxation approaches in Table 4 of our paper are erroneous. For reference, we approached the IS2RE task using two methods:
Direct: Directly predicting relaxed energies from initial states; or more broadly - trained on the IS2RE dataset (~460k).
Relaxation: Iteratively estimate atomic forces and update atomic positions until a relaxed state is reached. Report the energy of the found relaxed state; trained on the S2EF dataset (~134M).
The errors were a result of a bug later found in the denormalization of the model outputs in the custom script that was used to generate the final outputs. The corrected numbers are provided in the updated table below. Note, the Relaxation-based approaches now produce the best-performing numbers. We have updated the leaderboard with the new results. We will update the paper shortly with these results and another bug affecting relaxations with the CHCOH adsorbate and a few bulk surfaces (<1% of the overall dataset resulting in small changes in accuracy). Additionally, we plan to add an extra column to the IS2RE leaderboard to indicate whether entries were trained using the IS2RE dataset (direct) or S2EF dataset (relaxation) since these approaches are notably different.
We’re sorry for any trouble or confusion this error may have caused. Please reach out if you have any questions or concerns.
Hi - Entries marked with force-only + energy-only (i.e. DimeNet++ - force-only + energy-only) used a model trained on S2EF for forces only to run relaxations, followed with a model trained on S2EF for energy only, like you mentioned. Models missing that marking (SchNet, DimeNet++) used a single model trained on S2EF for energy+forces to run a relaxation and get energy predictions. While two models were necessary to improved DimeNet++ predictions, Table 4 of the SpinConv paper https://arxiv.org/pdf/2106.09575.pdf demonstrates its possible to get good results with a single model.
The problem is:
I ran the relaxations using pretrained DimeNetpp force-only model, and then used pretrained DimenetPP energy+forces to predict the energy. But I got huge (10^20) loss. From your answer I understand it shouldn’t be this way.
Also if I try to predict the energy of the real relaxed structure (setting data_object.pos = data_object.pos_relaxed) I got loss = nan. For some parts of the dataset getting huge (10^20) loss.
If I try to predict the is2re using pretrained IS2RE DimeNetpp I got loss as expected 0.5639.
So I don’t think I have any bug.
It’s a little hard to debug without knowing the exact procedure for each of those intermediate steps. I have a PR in progress to provide more clarity and utility scripts to what that procedure looks like - IS2RE relaxation scripts by mshuaibii · Pull Request #253 · Open-Catalyst-Project/ocp · GitHub. Note this is still a work in progress, but hopefully it can be of some help. Feel free to provide suggestions if there’s something in particular that would be useful.
Regarding energy predictions of the relaxed state, this is a little strange. If you can provide some reference/script to how exactly you did this I could be of more help. One thing I notice, however, is that changing pos to pos_relaxed doesn’t update the edge connectivity - leading to inaccurate predictions (and possibly related to your problem). You’ll need to set otf_graph=True to make sure edges are recomputed rather than using the ones stored in the LMDBs.
Absolutely forgot that connectivity of graph changed after the relaxations.
After I set otf_graph=True, both problems resolved.
Thank you for the help.
