Troubleshooting

Here are few know issue and how to troubleshoot them.

Pytorch not compiled with CUDA

Fix by reinstall pytorch pre-build with CUDA:

pip uninstall pytorch

or

conda uninstall torch

and follow instruction for Pytorch website on how to install current pytorch version with CUDA on your system.

TARDIS produce no output and no error message

This could be because TARDIS could not find any know object (e.g. membrane or microtubules) in your tomogram or micrograph.

You can try to reduce -ct <float; default 0.25> to 0.1 or 0.05. If this wont help, please contact Developers.

Error loading model: invalid load key, '<'

The full message looks like:

_pickle.UnpicklingError: invalid load key, '<'.

This means the model_weights.pth file is not a model checkpoint - it is an XML error page (starting with <) that was saved by a failed download, for example NoSuchKey or AccessDenied returned by S3. A genuine checkpoint is several MB to hundreds of MB; a corrupted one is usually a few hundred bytes.

Fix:

  1. Delete the cached weights and let TARDIS re-download them with internet access:

rm -rf ~/.tardis_em

  1. Or, if you pre-downloaded the weights manually, replace the corrupted file with a direct download (see Offline / air-gapped clusters below).

Recent TARDIS versions detect this automatically: a corrupted cached file is re-downloaded, and a failed download aborts with a clear error instead of being saved as a checkpoint.

Offline / air-gapped clusters

On clusters where compute nodes have no internet access, pre-download the weights on a login node and point TARDIS at them with -cch / -dch (or via the $TARDIS_CNN_MODELS / $TARDIS_DIST_MODELS environment variables).

The weights live in the public S3 bucket tardis-weigths:

  • CNN models: tardis_em/fnet_attn_32/<model>/<version>/model_weights.pth

  • DIST models: tardis_em/dist_triang/<model>/<version>/model_weights.pth

Option 1 - tardis_fetch (downloads one model at a time):

tardis_fetch -dir ./weights -mc actin_3d -md 3d

Option 2 - mirror the whole bucket with rclone (no AWS credentials needed):

mkdir -p weights

rclone config create tardis_s3 s3 \
  provider AWS \
  env_auth false \
  region us-east-1

rclone copy \
  --s3-no-check-bucket \
  --progress \
  tardis_s3:tardis-weigths/tardis_em \
  ./weights

Option 3 - download a single file directly, e.g. the 3D actin CNN model:

curl -L -o model_weights.pth \
  https://tardis-weigths.s3.us-east-1.amazonaws.com/tardis_em/fnet_attn_32/actin_3d/V_3/model_weights.pth

After downloading, sanity-check the file size (a few hundred bytes means the download failed and returned an error page):

ls -lh weights/**/model_weights.pth