-
Check if logrotate is enabled.
-
Check if the Nginx TLS certificate requires a password on startup.
Troubleshooting
The sections below covers common issues that may arise on a system and how to address them.
System
The table below provides guidance for resolving certain system-related issues that may arise.
Symptom | Reason | Verification | Next steps |
---|---|---|---|
Application shuts down regularly |
Nginx TLS config and logrotate conflict |
Choose one:
|
|
Unable to install or update rpm/deb packages using yum/apt |
System is not properly subscribed to SD Elements update server |
+-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Current
Loaded plugins: fastestmirror, presto, product-id, search-disabled-repos, *subscription-manager* Setting up Update Process Determining fastest mirrors Default_Organization_Centos_7_centosplus_x86_64 Default_Organization_Centos_7_extras_x86_64 Default_Organization_Centos_7_os_x86_64 Default_Organization_Centos_7_updates_x86_64 Default_Organization_Custom_7_custom_x86_64 Default_Organization_EPEL_6Server_x86_64 Default_Organization_EPEL_6_x86_64 Default_Organization_IUS_https_dl_iuscommunity_org_pub_ius_stable_CentOS_7_x86_64 Default_Organization_Katello_latest_client_el6_x86_64 Default_Organization_NGINX_centos_7_x86_64 Default_Organization_PostgreSQL_9_4_redhat_rhel-7-x86_64 Default_Organization_PostgreSQL_9_4_redhat_rhel-7Server-x86_64 Default_Organization_Puppet_el_6_dependencies_x86_64 Default_Organization_Puppet_el_6_pc1_x86_64 Default_Organization_Puppet_el_6_products_x86_64 |
|
Error running |
No outbound access to |
Run in offline mode: |
|
Nginx TLS/SSL error |
Check if Nginx certificate specifies the server’s fully qualified domain name (FQDN) |
Update certificate for server’s FQDN. Re-run reprovision. |
|
Malformed |
Check if |
Update the hosts file. Re-run reprovision. |
|
Limited diskspace |
Lingering PostgreSQL temp files |
Check the |
Remove the temp files |
Error running |
SDE downgraded SDE Admin during upgrade |
[root@hostname ~]# sde version Traceback (most recent call last): File "/usr/local/lib64/python3.6/site-packages/sde_admin/__init__.py", line 234, in <module> slugger = slugify.Slugify( AttributeError: module 'slugify' has no attribute 'Slugify' Failed to load sde_admin module |
Upgrade user=sc_user pass=mypassword sde_admin_version=2.2.14 umask 0002 && pip install --upgrade --force-reinstall sde-admin==${sde_admin_version} --index=https://${user}:${pass}@pypi.sdelements.com/pulp/python/web/prod/simple/ |
Upgrades
The table below provides steps to resolve upgrade issues.
Symptom | Reason | Verification | Next steps |
---|---|---|---|
Connection problem to |
No outbound network access from the server to |
Run commands: ping updates.sdelements.com ping anvil.sdelements.com |
|
Network access limited by local firewall |
Run command: iptables -nL |
|
|
Network access limited by firewall |
Run commands: tracepath updates.sdelements.com/443 tracepath anvil.sdelements.com/443 |
|
|
Update fails |
An unexpected issue encountered in the updater. |
Upgrade the SD Elements updater and try the upgrade again. |
Check |
Issue Tracker, Scanner & LDAP integration
The table below provides troubleshooting guidance for issues regarding integration with an Issue Tracker, scanner or LDAP server.
Symptom | Reason | Verification | Next steps |
---|---|---|---|
Invalid server or server unreachable |
Connection details are invalid |
Verify the connection details are correct. |
Update the connection with the correct information and retry. |
No network access to |
Run command: ping server.host.name |
|
|
Network access limited by local firewall |
Run command: iptables -nL |
|
|
Network access limited by firewall |
Run command: tracepath server.host.name/port |
|
|
Network access is limited by a transparent proxy |
A transparent proxy may be at issue if outbound network access is already confirmed for other external systems but not for this server. |
Transparent proxies allow companies to control traffic without burdening systems with configuration. Contact the IT team for details and request a whitelist to the desired endpoint, if needed. |
|
Network access is limited by an IPS (Intrusion Prevention System) |
Check with IT team if traffic to |
Investigate the cause for rule being triggered. Request an exception for the specific server. |
|
TLS/SSL validation error |
HTTPS connection fails certificate validation |
||
Connection relies on a proxy that rewrites TLS/SSL certificates or its own certificate is untrusted. |
Check that the proxy’s certificate or its CA certificate is trusted by the system: Validate TLS/SSL connection to the proxy. |
||
TLS/SSL connection error |
HTTPS connection to server requires Server Name Indication (SNI) support |
Contact SD Elements product team to prioritize SNI support |
|
HTTPS connection fails due to cipher or protocol error |
Investigate whether the target server supports minimum TLS security settings. For example, SSLv3 is not supported. |
||
Jobs stuck or not working |
Celery needs a restart |
Application shows jobs stuck in status "Waiting…" for more than 10 minutes |
On the SSH console run: sde supervisor restart all |
Inconsistent connection |
DNS issue |
If connection to a server fails intermittently, the problem may be due to a flaky DNS lookup. |
Add an entry to |
Job unexpectedly fails |
Integration issue or unsupported server |
Examine celery logs for the error. |
|
Timeout reached |
Examine celery logs for a |
||
Integration server error |
Examine celery logs. |
|
|
LDAP SSO error |
Use the in-app troubleshooting mechanism. |
|
|
Missing LDAP configuration |
Examine |
Update |
Capture detailed integration logs
Diagnosing integration issues is aided greatly by detailed logs between SD Elements and the other server. Follow the steps below to collect verbose logs for a problematic integration.
-
SSH credentials for sde_admin or sudo access.
-
Application Super User access.
-
Login to the SD Elements web application as a Super User.
-
Open the problematic integration connection.
-
Enable option Debug Mode.
-
Access the SD Elements server SSH console as sde_admin.
-
Run command:
sde manage_django run_session_capture_server 2> /docs/sde/log/debug_integration.log
-
Run the problematic integration until it completes.
-
Disable option Debug Mode on the integration.
-
Cancel the
run_session_capture_server
command by entering Ctrl-C.
The full integration logs are captured in file debug_integration.log
.
Credentials are stored as cleartext in the log file. Remove the file from the system as soon as possible. |
Modify the application job timeout
Integrations with Issue Tracker systems, scanning tools, and LDAP servers are run by the Celery process. By default these jobs time out after 10 minutes.
To modify the job timeout to 15 minutes, for example, follow the steps below.
-
SSH credentials for sde_admin
-
Access the SD Elements server SSH console as sde_admin.
-
Update file
/docs/sde/local_settings
set:CELERY_JOB_TASK_SOFT_TIME_LIMIT = 15 * 60
-
Save the file and run:
sde supervisor restart all sde apache restart
New jobs are configured to expire after 15 minutes.
Modify the GitHub API delay rate
The default delay between GitHub issue creation and updates is 20 seconds. This delay is necessary to adhere to GitHub’s secondary rate limiting for certain content creation endpoints.
If you are a GitHub Enterprise client and have rate limiting disabled, or if the default timeout is insufficient for syncing all project Countermeasures, follow the steps below to modify the timeout value.
-
SSH credentials for sde_admin.
-
Access the SD Elements server SSH console as sde_admin.
-
Open the file
/docs/sde/local_settings
and add or update the following line:SDETOOLS_GITHUB_API_REQUEST_DELAY_SECONDS = 30
-
Save the file and run:
sde supervisor restart all sde apache restart
The example increases the timeout from 20 seconds to 30 seconds. Use a value that works best for your use case.
LDAP
See LDAP Troubleshooting.
SAML
See SAML Troubleshooting.
Open a support case using the support portal
If the system is acting abnormally, reach out to SD Elements Support with details about the issue. You will receive a response within 3 business hours.
-
A screenshot from the application and/or copy of the logs containing the error
-
Open the support portal https://support.securitycompass.com
-
Click on Submit a request
-
Enter your details:
-
Email address: A way for the support team to respond to you directly.
-
Subject: A brief description of the issue
-
Description: The issue you experienced as well as steps to reproduce. include your system versions.
-
Attachments: Screenshots, log files, or other information helpful to better understand and diagnose your issue.
-
-
Click Submit.
A new support ticket is created for your issue. You will receive an email soon as confirmation.
Open a support case using email
If the system is acting abnormally, reach out to SD Elements Support with details about the issue. You will receive a response within 3 business hours.
-
A screenshot from the application and/or copy of the logs containing the error
-
Compose a new email to sdesupport@securitycompass.com
-
Subject: A brief description of the issue
-
Body: The issue you experienced as well as steps to reproduce. include your system versions.
-
Attachments: Screenshots, log files, or other information helpful to better understand and diagnose your issue.
-
-
Click Send.
A new support ticket is automatically created for your issue. You will receive an email soon as confirmation.