Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 23 additions & 64 deletions content/uw-research-computing/docs/htc-known-issues.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,32 @@ category: Troubleshooting

This page documents some common and known issues encountered on the HTC system. While this page can be beneficial in troubleshooting, it does not contain a comprehensive list of errors.

Visit our [Get Help](/uw-research-computing/get-help/) page to find more resources for troubleshooting.
<ButtonRow>
<Button href="/uw-research-computing/get-help">Don't see an answer to your question? Get help</Button>
</ButtonRow>

{/* Removing since this seems to be fixed
<hr width="100%" size="2">
## General

<h3 style="color:#c5050c" id="general-when-submitting-a-job-it-doesnt-run-goes-on-hold-and-shows-the-error-job-credentials-are-not-available">[General] When submitting a job, it doesn't run / goes on hold and shows the error "Job credentials are not available".</h3>
<MiniBar>
### When submitting a job, it doesn't run / goes on hold and shows the error "Job credentials are not available".
</MiniBar>

#### Cause:
This is a complicated bug that can strike randomly. We're working on a fix.
#### Solution:
To work around this issue, run the following command on the access point before resubmitting the job.

<div class="md-term">
<Terminal>

```
echo | condor_store_cred add-oauth -s scitokens -i -
```

</div>

*/}
<hr width="100%" size="2" />
</Terminal>

<h3 style={{ color: '#c5050c' }} id="general-my-job-exits-with-an-illegal-instructions-error">[General] My job exits with an "Illegal instructions" error.</h3>
<MiniBar>
### My job exits with an "Illegal instructions" error.
</MiniBar>

#### Cause:
The instructions for communicating with the processor does not match between software and hardware.
Expand All @@ -46,23 +48,11 @@ Then resubmit and check if this resolves the issue.

<hr width="100%" size="2" />

<h3 style={{ color: '#c5050c' }} id="general-i-used-generative-ai-to-create-my-submit-file-and-the-job-is-stuck-on-idle">[General] I used generative AI to create my submit file and the job is stuck on "Idle".</h3>

#### Cause:
Generative AI (such as Copilot, ChatGPT, etc.) may be inaccurate.
In our experience, generative AI almost always gets something wrong when it comes to an HTCondor submit file.
This is because HTCondor has significantly changed since its creation over 40 years ago, and is highly configurable.
This means that while there is a lot of information online in forums and site-specific documentation, the information is not always relevant to our local HTC system.
As such, submit files created by generative AI are usually not applicable to our HTC system.
#### Solution:
The CHTC website provides documentation - including example submit files - for how to submit jobs on the HTC system.

* Learn the basics of job submission here: [chtc.cs.wisc.edu/uw-research-computing/htcondor-job-submission](https://chtc.cs.wisc.edu/uw-research-computing/htcondor-job-submission).
* For submitting jobs requesting GPUs, see our dedicated GPU guide here: [chtc.cs.wisc.edu/uw-research-computing/gpu-jobs](https://chtc.cs.wisc.edu/uw-research-computing/gpu-jobs).

<hr width="100%" size="2" />
## Containers

<h3 style={{ color: '#c5050c' }} id="container-when-building-an-apptainer-apt-commands-in-the-post-block-fail-to-run">[Container] When building an Apptainer, "apt" commands in the %post block fail to run.</h3>
<MiniBar>
### When building an Apptainer, "apt" commands in the %post block fail to run.
</MiniBar>

#### Example error message:

Expand All @@ -88,10 +78,9 @@ From: ubuntu:22.04
```
We also recommend using the `-y` option to prevent installation from hanging due to interactive prompts.


<hr width="100%" size="2" />

<h3 style={{ color: '#c5050c' }} id="container-after-submitting-a-job-with-a-docker-container-it-goes-on-hold-with-a-message-like-cannot-pull-image-userrepotag">[Container] After submitting a job with a Docker container, it goes on hold with a message like, "Cannot pull image user/repo:tag".</h3>
<MiniBar>
### After submitting a job with a Docker container, it goes on hold with a message like, "Cannot pull image user/repo:tag".
</MiniBar>

#### Cause:
The Docker container is likely built on an Apple computer using an ARM processor, which is incompatible with Linux machines.
Expand All @@ -107,9 +96,9 @@ docker build --platform linux/amd64 .
</Terminal>


<hr width="100%" size="2" />

<h3 style={{ color: '#c5050c' }} id="container-my-interactive-apptainer-job-is-failing-with-the-error-message-cant-open-master-pty-bad-file-descriptor">[Container] My interactive Apptainer job is failing with the error message, "Can't open master pty Bad file descriptor". </h3>
<MiniBar>
### My interactive Apptainer job is failing with the error message, "Can't open master pty Bad file descriptor".
</MiniBar>

#### Cause:
Interactive Apptainer jobs are incompatible with CentOS7 machines.
Expand All @@ -121,36 +110,6 @@ requirements = (OpSysMajorVer > 7)

<hr width="100%" size="2" />

<h3 style={{ color: '#c5050c' }} id="gpu-my-gpu-job-has-been-in-the-queue-for-a-long-period-of-time-and-is-not-starting">[GPU] My GPU job has been in the queue for a long period of time and is not starting. </h3>

#### Cause:
To use our shared use GPU machines, you must opt into the GPU Lab.
#### Solution:
To your submit file, add the following line and resubmit:
```
+WantGPULab = true
```

<br />
<hr width="100%" size="2" />

<h3 style={{ color: '#c5050c' }} id="general-python-my-job-has-an-error-that-mentions-homenetid">[General, Python] My job has an error that mentions <code>/home/netid/</code> or the root directory <code>/</code>. </h3>

#### Cause:
Sometimes programs assume that they can write into the `/home` or root `/` directory. However, these directories are not writeable on the Execution Points.

#### Solution:
To your executable (`.sh` script), add the following line and resubmit:
```
export HOME=$PWD
```
This will change the `HOME` environment variable to the working directory on the Execution Point.

For any items referencing the root directory `/`, try the solution suggested above. If that does not fix the issue, you will need to find where your code or program is referencing the root directory and change it to the current working directory. Please [email us or drop by office hours](/uw-research-computing/get-help/) if you need assistance with this.

<br />
<hr width="100%" size="2" />


## Can't find your issue?

Visit our [Get Help](/uw-research-computing/get-help/) page.
Loading