Table of Contents
- 1. tty - teletype or terminal
- 2. shell
- 2.1. variables
- 2.2. Enveronmental Variables
- 2.3. Wildcard
- 2.4. {} Curly Braces
- 2.5. {} Brace Expansion
- 2.6. Extended globs
- 2.7. stdin(0), stdout(1), and stderr(2), pipes
- 2.8. String processing
- 2.9. Process Control
- 2.10. Programming
- 2.10.1. shebang
- 2.10.2. pros and cons
- 2.10.3. Hello World.sh, scriptinga
- 2.10.4. arguments
- 2.10.5. array
- 2.10.6. comparision Posix
- 2.10.7. numbers comparision
- 2.10.8. strings comparision
- 2.10.9. if
- 2.10.10. loop
- 2.10.11. file comparision
- 2.10.12. functions
- 2.10.13. debug
- 2.10.14. exit status
- 2.10.15. generate random
- 2.10.16. documentation
- 2.10.17. best practices
- 2.10.18. export
- 2.10.19. null command
- 2.10.20. multiline comment
- 2.10.21. read
- 2.11. set builtin
- 2.12. ls
- 2.13. pushd/popd and dirs
- 2.14. sh script
- 2.15. source & ./ & exec
- 2.16. ariphmetic
- 2.17. execute command
- 2.18. xargs
- 2.19. history save with time
- 2.20. id -u user
- 2.21. diff
- 2.22. date
- 2.23. ps
- 2.24. top
- 2.25. pgrep
- 2.26. vmstat
- 2.27. pmap
- 2.28. find
- 2.29. dd
- 2.30. access control to files
- 2.31. for external storages
- 2.32. filesystem block size
- 2.33. tar
- 2.34. compression
- 2.35. curl
- 2.36. du - file space usage
- 2.37. man, info, help
- 2.38. less
- 2.39. date
- 2.40. tee
- 2.41. time
- 2.42. lsfd
- 2.43. fuser
- 2.44. administration commands
- 2.45. USECASES
- 2.46. links
- 3. Bash new features
- 4. tmux
- 5. SSH
- 5.1. config files
- 5.2. permissions
- 5.3. authentication types
- 5.4. usage
- 5.5. commands
- 5.6. 1) ssh server - password
- 5.7. 2) ssh server - generate key - public key and private key
- 5.8. 3) sftp server
- 5.9. 4) X11 Forwarding
- 5.10. 5) sshfs - client
- 5.11. TODO Internet over socks
- 5.12. DNS over socks
- 5.13. ssh-agent
- 5.14. agent forwarding problem
- 5.15. ssh-copy-id
- 5.16. inside of ssh
- 5.17. old
- 5.18. githib ~/.ssh/config
- 5.19. key type
- 5.20. security
- 5.21. Port Forwarding/Tunneling
- 5.22. Reverse SSH Tunnel
- 5.23. USE CASES
- 5.24. user with limited permissions
- 5.25. links
- 5.26. ssh over socks5
- 5.27. problems
- 5.28. ssh as a tor hidden service
- 5.29. ssh systemd service
- 5.30. rsync over ssh
- 6. jq - JSON parser
- 7. print JSON
;-- mode: Org; fill-column: 110;--
1. tty - teletype or terminal
1.1. keys
- https://ss64.com/bash/syntax-keyboard.html
- $ info readline
- https://git.savannah.gnu.org/cgit/readline.git/tree/doc/rluserman.info
- man termios
It's basically a device (implemented in software nowadays)
Ctrl+- undo
Ctrl+[ = Escape Ctrl+I = Tab Ctrl+M = Enter
Alt+/ = autocompletion for path Ctrl+J = Newline Ctrl+v tells the terminal to not interpret the following character, so Ctrl+v Ctrl-I will display a tab character, similarly Ctrl+v ENTER will display the escape sequence for the Enter key: ^M
Ctrl + C SIGINT, Break command Ctrl + Z in background, commands: jobs, fg, bg Ctrl + D SIGTSTP, logout Ctrl + \ SIGQUIT
Ctrl + XX toggle cursor from current position to begining and back Alt+U/L to UPPER/LOWER case word Alt+C upper first letter of word
Ctrl/Alt + D: Delete one char/to end of word starting at the cursor
Ctrl + u/k erase line to begining/end of line Ctrl + w cut word before cursor Alt + k cut word before cursor Alt + b cut word after cursor Ctrl+y yank
Ctrl+N/P loop commands Ctrl+S Stops all output on-screen (XOFF). Ctrl+Q Turns all output stopped on-screen back on (XON). Also, closes an application window.
!! Repeats the line last entered at the shell. !$ Repeats the last argument for the command last used. See history command for previous commands.
Alt+SysRq+r Unraw Take control of keyboard back from X. 4 Alt+SysRq+e Terminate Send SIGTERM to all processes, allowing them to terminate gracefully. 64 Alt+SysRq+i Kill Send SIGKILL to all processes, forcing them to terminate immediately. Alt+SysRq+s Sync Flush data to disk. 16 - Alt+SysRq+u Unmount Unmount and remount all filesystems read-only. 32 - Alt+SysRq+b Reboot Reboot 128 Alt+SysRq+k SAK Alt+SysRq+f OOM kill Alt+SysRq+j Thaw
X Ctrl+Alt+F1, F2, F3, … Switch to n-th virtual console If it does not work, try Ctrl+Alt+Fn+F…. Shift+Insert Paste text from the PRIMARY buffer
M-O in console activates when some key is pressed and holded for some time
1.1.1. special Bash keys
https://git.savannah.gnu.org/cgit/bash.git/tree/bashline.c?id=3ba697465bc74fab513a26dea700cc82e9f4724e#n957 https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html C-x ~ - bash_possible_username_completions C-x @ - bash_complete_hostname C-x C-$ - bash_complete_variable C-x C-! - bash_complete_command C-x / - bash_complete_filename C-x C-e - Bash attempts to invoke $FCEDIT, $EDITOR, and emacs as the editor, in that order. C-x C-v - Bash version C-x C-r - re-reads this init file, thus incorporating any changes that you might have made to it.
Alt+number - repeat next key
C-_ or C-x C-u Undo the last editing command. You can undo all the way back to an empty line.
1.1.2. History:
Ctrl + r Recall the last command including the specified character(s). searches the command history as you type. Ctrl + Shift + r - reverse reverse-i-search Equivalent to : vim ~/.bash_history. Ctrl + p Previous command in history (i.e. walk back through the command history). Ctrl + n Next command in history (i.e. walk forward through the command history).
Ctrl + s Go back to the next most recent command. (beware to not execute it from a terminal because this will also launch its XOFF). Ctrl + o Execute the command found via Ctrl+r or Ctrl+s Ctrl + g Escape from history searching mode !! Repeat last command !n Repeat from the last command: args n e.g. !:2 for the second argumant. !n:m Repeat from the last command: args from n to m. e.g. !:2-3 for the second and third. !n:$ Repeat from the last command: args n to the last argument. !n:p Print last command starting with n !string Print the last command beginning with string. !:q Quote the last command with proper Bash escaping applied. Tip: enter a line of Bash starting with a # comment, then run !:q on the next line to escape it. !$ Last argument of previous command. ALT + . Last argument of previous command. !* All arguments of previous command. ^abc^def Run previous command, replacing abc with def
1.1.3. Process control:
Ctrl + C Interrupt/Kill whatever you are running (SIGINT). Ctrl + l Clear the screen. Ctrl + s Stop output to the screen (for long running verbose commands). Then use PgUp/PgDn for navigation. Ctrl + q Allow output to the screen (if previously stopped using command above). Ctrl + D Send an EOF marker, unless disabled by an option, this will close the current shell (EXIT). Ctrl + Z Send the signal SIGTSTP to the current task, which suspends it. To return to it later enter fg 'process name' (foreground).
1.1.4. Moving the cursor:
Ctrl + a | Go to the beginning of the line (Home) |
Ctrl + e | Go to the End of the line (End) |
Ctrl + p | Previous command (Up arrow) |
Ctrl + n | Next command (Down arrow) |
Alt + b | Back (left) one word |
Alt + f | Forward (right) one word |
Ctrl + f | Forward one character |
Ctrl + b | Backward one character |
Ctrl + xx | Toggle between the start of line and current cursor position |
1.1.5. Editing:
Ctrl + L | Clear the Screen, similar to the clear command |
Alt + Del | Delete the Word before the cursor. |
Alt + d | Delete the Word after the cursor. |
Ctrl + d | Delete character under the cursor |
Ctrl + h | Delete character before the cursor (Backspace) |
Ctrl + w | Cut the Word before the cursor to the clipboard. |
Ctrl + k/u | Cut the Line after/before the cursor to the clipboard. |
Alt + t | Swap current word with previous |
Ctrl + t | Swap the last two characters before the cursor (typo). |
Esc + t | Swap the last two words before the cursor. |
ctrl + y | Paste the last thing to be cut (yank) |
Alt + u | UPPER capitalize every character from the cursor to the end of the current word. |
Alt + l | Lower the case of every character from the cursor to the end of the current word. |
Alt + c | Capitalize the character under the cursor and move to the end of the word. |
Alt + r | Cancel the changes and put back the line as it was in the history (revert). |
ctrl + _ | Undo |
1.2. how it works
https://unix.stackexchange.com/questions/4126/what-is-the-exact-difference-between-a-terminal-a-shell-a-tty-and-a-con https://dev.to/napicella/linux-terminals-tty-pty-and-shell-192e https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch18.html https://en.wikipedia.org/wiki/POSIX_terminal_interface https://habr.com/ru/post/460257/ https://medium.com/securitythread/linux-terminal-subsystem-d000475c8dd www.embeddedlinux.org.cn/essentiallinuxdevicedrivers/final/ch06lev1sec4.html
tty - teletypewriter https://en.wikipedia.org/wiki/Teleprinter
- terminal - became a computer program in the kernel which would send characters directly to the TTY
driver, read from it and print to the screen.
- is a character-oriented device, comprising streams of characters received from and sent to the device, dev/tty - device which implements a number of additional commands (ioctls) beyond read and write.
- consist of:
- character device interface ( open, read, write, close …)
- Line discipline
- driver - keyboard IO, display
- may be: kernel program or pseudo terminal or serial device (old)
- communicate with the terminal through terminal character device file
- terminal emulator - the kernel program emulate the physical terminal device. it listens for events coming from the keyboard and sends it down to the driver
- pseudo-terminal (pty) - provided by terminal emulators - it's not a kernel program but one that runs in the user land.
- System console - interface more
- physical device consisting of a keyboard and a screen
- text entry and display device for system administration messages
- Virtual console/virtual terminal - Ctrl+Alt+F1234
- On each virtual terminal, a getty process is run, which in turn runs /bin/login to authenticate a user.
- supported at the Linux kernel level
- are emulated text terminals, using the keyboard and monitor of a personal computer or workstation
- terminal monitor program spawns (sets TERM env - Specifies the terminal type.)
- may be:
- getty TERM=linux
- SSH server ( to the same terminal type as the SSH client)
- software terminal emulator, using a pseudoterminal ( sets the TERM environment variable to
specify the type of terminal that it is emulating)
- GNU Screen program sets screen as the terminal type.
- may be:
- shell - user space app
- escape code - . Certain sequences of bytes, most starting with an ASCII escape character and a
bracket character, are embedded into text to control cursor location, color, font styling, and
other options on video text terminals and terminal emulators.
- man console_codes - Linux console escape and control sequences
Terminal emulator(get keys and draw gui) -> PTY master -> tty driver(with line discipline) -> pty slave -> bash
По умолчанию TTY устройство работает в каноничном режиме с включенным эхо (echoing). Эхо — это отображение введенных вами символов на экране.
GNU Readline - library that provides in-line editing and history capabilities - supports both Emacs and vi editing modes
terminfo - used by TERM enveronmental variable. which strings to send for clear screen, move cursor, etc.
- database: /usr/share/terminfo/*/* - /etc/terminfo/*/*
- default = /etc/terminfo/l/linux
agetty - opens a tty port
1.3. terminal related commands:
- who
- w
- last
- tty
- ps
- stty - change terminal settings, prints baud rate, line discipline, and deviations from stty sane
- stty -a - get all
- stty icanon -echo - disable echo
- stty raw echo
- stty <control-character> <string>
- stty erase 0
- reset - reset terminal after cat binary file
- infocmp - compare or print out terminfo descriptions
1.4. escape codes
The escape codes received from the terminal signify things such as function key, arrow key, and other special keystrokes (home key, end key, help key, PgUp key, PgDn key, insert key, delete key, and so forth)
1.5. TODO POSIX terminal interface
1.6. pseudo-teletypes (PTS)
The multiplexor is the master, and the PTS are the slaves. The multiplexor is addressed by the kernel through the device file located at /dev/ptmx.
1.7. history links
1.8. tty scroll
- tmux
1.8.1. screen
Ctrl-A, Esc - now Page Up, Page down Esc - exit
without:
- Ctrl+z - background
- $ fg - front
Scroll: Ctrl-A, Esc - now Page Up, Page down
- screen - start
- Ctrl+a+c create new
- Ctrl+a+" list
- Ctrl+a+d deattach
- Ctrl+a+b next window
- Ctrl+a to last used
- Ctrl+a+0 switch to 0 screen
- Shift+a - Set title for window.
Copy/paste
- Ctrl-A, Esc
- Enter, Enter
- Ctrl-A, ] - paste
1.9. clipboard
- emerge x11-misc/xsel
- bind '"\C-p": "\C-e\C-u xsel <<"EOF"\n\C-y\nEOF\n\C-y"'
- emerge x11-misc/xclip
- cat a | xclip -sel clip
1.10. keymaps, dvorak
ubuntu: apt install console-keymaps
- loadkeys /usr/share/keymaps/i386/dvorak/dvorak.kmap.gz
1.11. ASCII
0 ^@ \0 Null 1 ^A Start of Heading 2 ^B Start of Text 3 ^C End of Text 4 ^D End of Transmission 5 ^E Enquiry 6 ^F Acknowledgement 7 ^G \a Bell 8 ^H \b Backspace[e][f] 9 ^I \t Horizontal Tab[g] 10 ^J \n Line Feed 11 ^K \v Vertical Tab 12 ^L \f Form Feed 13 ^M \r Carriage Return[h] 14 ^N Shift Out 15 ^O Shift In 16 ^P Data Link Escape 17 ^Q Device Control 1 (often XON) 18 ^R Device Control 2 19 ^S Device Control 3 (often XOFF) 20 ^T Device Control 4 21 ^U Negative Acknowledgement 22 ^V Synchronous Idle 23 ^W End of Transmission Block 24 ^X Cancel 25 ^Y End of Medium 26 ^Z Substitute 27 ^[ \e[i] Escape[j] 28 ^\ File Separator 29 ^] Group Separator 30 ^^ Record Separator 31 ^_ Unit Separator 32 space 33 ! Exclamation mark 34 " Double quotes (or speech marks) 35 # Number sign 36 $ Dollar 37 % Per cent sign 38 & Ampersand 39 ' Single quote 40 ( Open parenthesis (or open bracket) 41 ) Close parenthesis (or close bracket) 42 * Asterisk 43 + Plus 44 , Comma 45 - Hyphen-minus 46 . Period, dot or full stop 47 / Slash or divide 48 0 49 1 50 2 51 3 52 4 53 5 54 6 55 7 56 8 57 9 58 : Colon 59 ; Semicolon 60 < Less than (or open angled bracket) 61 = Equals 62 > Greater than (or close angled bracket) 63 ? Question mark 64 @, `, @ At sign 65 A 66 B 67 C 68 D 69 E 70 F 71 G 72 H 73 I 74 J 75 K 76 L 77 M 78 N 79 O 80 P 81 Q 82 R 83 S 84 T 85 U 86 V 87 W 88 X 89 Y 90 Z 91 [ Opening bracket 92 \, ~, \ Backslash 93 ] Closing bracket 94 ↑, ^ Caret - circumflex 95 ←, _ Underscore 96 @, ` Grave accent 97 a 98 b 99 c 100 d 101 e 102 f 103 g 104 h 105 i 106 j 107 k 108 l 109 m 110 n 111 o 112 p 113 q 114 r 115 s 116 t 117 u 118 v 119 w 120 x 121 y 122 z 123 { Opening brace 124 ACK, ¬, | Vertical bar 125 } Closing brace 126 ESC, |, ~ Equivalency sign - tilde 127 DEL Delete
sessions:
- tmux new-session -d top
- tmux ; C-b ( - switch to background session with top
- C-b ) ; C-d - deattach from background session
2. shell
You may indent or not indent however you like
2.1. variables
${ varname } is general syntax for $variable
- $_ At shell startup, set to the pathname used to invoke the shell or shell script being executed
- $1 - first parameter
- $#: How many command line parameters were passed to the script.
- $?: The exit status of the last process to run.
- $$: The Process ID (PID) of the current script.
- $* is a single string that consists of all of the positional parameters, separated by the first character in the environment variable IFS
- $@: All the command line parameters passed to the script. Cat be set: $set – 1 2 3
- $USER: The username of the user executing the script.
- $HOSTNAME: The hostname of the computer running the script.
- $SECONDS: The number of seconds the script has been running for.
- $RANDOM: Returns a random number.
- $LINENO: Returns the current line number of the script.
- IFS (internal field separator), which is a space, TAB, and NEWLINE by default.
- env | less # environment variables
my=Dave my="Dave Green" file_count=$(ls $folder_to_count | wc -l) # miltiline
export mt_var # to save variables to outside of the script
multiline variable (heredoc):
method1="This is line no.1\nThis is line no.3\nThis is line no.3" echo -e $method1 read -r -d '' MULTI_LINE_VAR << EOM Print line 1. Print line 2. Print line 3 EOM
2.2. Enveronmental Variables
- ( set -o posix ; set ) | less
- env
- printenv - same as env
2.3. Wildcard
2.3.1. Standard Wildcards (globbing patterns)
globbing for pathnames
used by nearly any command
- man 7 glob
- ^$ - begin and end
- . - any single character
- \ - escaping
- [] - character in range
2.3.2. advanced
move all except one:
- shopt -s extglob
- mv !(fileOne) ~/path/newFolder
ls hello.???
ls [a-p0-5]*.* # dot means dot
[!…] or [^…] which matches any character except the ones in the set.
brace expansion {}
- it actually generates all the permutations of the pattern you specify and then performs wildcard expansion on the results.
- echo d{a,e,i,u,o}g => dag deg dig dug dog
- echo d{a..z}g => dag dbg dcg ddg deg dfg dgg dhg dig djg dkg dlg dmg dng dog dpg dqg drg dsg dtg dug dvg dwg dxg dyg dzg
- echo {cat,dog} => cat dog
- echo {cat,d*} => cat dawg dg dig dog doug dug
- echo {j{p,pe}g,png} => jpg jpeg png
- echo .{mp{3..4},m4{a,b,p,v}} => .mp3 .mp4 .m4a .m4b .m4p .m4v
2.3.3. classes
- [:alpha:] It is used to match with any uppercase and lowercase letter and equivalent to [a-zA-Z].
- [:digit:] It is used to match with any digits and is equivalent to [0-9].
- [:alnum:] It is used to match any alphabet and digit. It is equivalent to [a-zA-Z].
- [:upper:] It is used to match with uppercase latter only and equivalent to [A-Z]
- [:lower:] It is used to match with uppercase latter only and equivalent to [a-z]
- [:blank:] It is used to match with space and tab.
- [:print:] It is used to match with printable characters.
- [:cntrl:] It is used to match with non-printable characters.
- [:space:] It is used to match with while-space.
- [:xdigit:] It is used to match with hexadecimal digits.
- [:ascii:] It is used to match with ASCII characters.
- [:word:] It is used to alphanumeric characters including underscore(_).
2.3.4. links
- linux wildcard https://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm
2.4. {} Curly Braces
$ echo {0..10} 0 1 2 3 4 5 6 7 8 9 10 echo a{d,c,b}e ade ace abe echo {a..z}{a..z} aa ab ac ad ae af ag ah ai aj ak al am an ao ap aq ar as at au av aw ax ... month=("Jan" "Feb" "Mar" "Apr") echo ${month[3]} Apr # { ... } to group the output from several commands into one big blob { echo "I found all these PNGs:"; find . -iname "*.png"; echo "Within this bunch of files:"; ls; } > PNGs.txt # Parameter Expansion: parameter=22 && echo $0 # parent command paramters bash parameter=22 && echo $parameter 22 parameter=22 && echo ${parameter} 22 parameter=abcd && echo ${parameter:1} # get [1:] ${parameter:offset:length} bcd a=213 && p=a && echo ${!p} # indirection 213 echo ${!#} # "#" stores the number of arguments !#=0 array=(2 1 3) && ${array[@]} == ${array[*]} # all elements bash p=abc && echo ${p^} # The ^ operator modifies the first character to uppercase, the , operator to lowercase Abc p=abc && echo ${p^^} ABC p=ABC && echo ${p~} # change chase aBC echo ${BASH*} # ${!PREFIX*} or ${!PREFIX@} variable names beginning with the string PREFIX BASH BASH_ARGC BASH_ARGV BASH_COMMAND BASH_LINENO BASH_SOURCE BASH_SUBSHELL BASH_VERSINFO BASH_VERSION # Substring removal: a="Hello World!" && echo Goodbye${a#Hello} # a chunk from the beginning of a variable Goodbye World! a="Too longgg" && echo ${a%gg} # chops off the last two gg and prints “Too long“. Too long p=123 && echo ${p%??} # remove 23 1 # "#" will try to remove the shortest text matching the pattern, while "##" tries to do it with the longest text matching. MYSTRING="Be liberal in what you accept, and conservative in what you send" && echo ${MYSTRING#*in} what you accept, and conservative in what you send MYSTRING="Be liberal in what you accept, and conservative in what you send" && echo ${MYSTRING##*in} what you send Get name without extension ${FILENAME%.*} ⇒ bash_hackers.txt Get extension ${FILENAME##*.} ⇒ bash_hackers.txt Get directory name ${PATHNAME%/*} ⇒ /home/bash/bash_hackers.txt Get filename ${PATHNAME##*/} ⇒ /home/bash/bash_hackers.txt echo "${array[@]%is}" # Simple example, removing a trailing is from all array elements (on expansion): ⇒ Th a text (it was: This is a text) # Search and replace ${PARAMETER/PATTERN/STRING} # first occurrence ${PARAMETER//PATTERN/STRING} # all occurrence ${PARAMETER/PATTERN} # remove ${PARAMETER//PATTERN} p=abc && echo ${#p} # length 3 # Use a default value p= && echo ${p:-aaa} # if unset or empty aaa p= && echo ${v:-aaa} # if unset aaa p= && echo ${p:=aaa} # p=aaa now aaa p= && echo ${p=aaa} # p=aaa now dont print "" p=123 && echo ${p:+aaa} # is set and not empty. but ignore p value aaa p= && echo ${p:+aaa} "" p= && echo ${p+aaa} # is set (empy or not). but ignore p value aaa p= && echo ${p:?aaa} # if not set or empty bash: p: aaa p= && echo ${p:?aaa} # if not set or empty bash: p: aaa p= && echo ${p?aaa} # if not set ""
2.5. {} Brace Expansion
- ls *{txt,log} list all files ending with txt or log in the current directory
- cp ~/projects/adders/verilog/{half_,full_}adder.v . copy half_adder.v and full_adder.v to current directory
- mv story.txt{,.bkp} rename story.txt as story.txt.bkp
- cp story.txt{,.bkp} to create bkp file as well retain original
- mv story.txt{.bkp,} rename story.txt.bkp as story.txt
- mv story{,_old}.txt rename story.txt as story_old.txt
- touch file{1..4}.txt same as touch file1.txt file2.txt file3.txt file4.txt
- touch file_{x..z}.txt same as touch file_x.txt file_y.txt file_z.txt
- rm file{1..4}.txt same as rm file1.txt file2.txt file3.txt file4.txt
- echo story.txt{,.bkp} displays the expanded version 'story.txt story.txt.bkp' , useful to dry run before executing actual command
2.6. Extended globs
- ?(pattern-list) Matches zero or one occurrence of the given patterns
- *(pattern-list) Matches zero or more occurrences of the given patterns
- +(pattern-list) Matches one or more occurrences of the given patterns
- @(pattern-list) Matches one of the given patterns
- !(pattern-list) Matches anything except one of the given patterns
sheck if enabled:
$ shopt extglob extglob on
ex.
$ ls !(*.txt) main.c math.h power.log
2.7. stdin(0), stdout(1), and stderr(2), pipes
standard streams == file descriptors
Commands has:
- 0 <- terminal
- 1 -> terminal
- 2 -> terminal
2.7.1. pipeline
- '|&' equal to '2>&1 |' std out+err to std input
- | - pipe stdout only
- command >(command1) >(command2g) - pipe to two and more processes
- command 2> >(command) >c - pipe
- (command1 ; command2 ; command3) | command4
2.7.2. redirection
- redirect stdout to a file ls > a.txt
- redirect stderr to a file 2> a.txt
- redirect stdout to a stderr ls 1>&2 >&2
- redirect stderr to a stdout ls 2>&1
- redirect stderr and stdout to a file ls &> /dev/null
- redirect stderr and stdout to stdout &>&1
- redirect stderr and stdout to stderr &>&2
>&2 means copy file descriptor 2 to 1 both will refer to the 2
more:
- command 3> file # open new file descriptor
- command < file # standard input from file
- tee a b c - redirection to multiple files (>a >b >c - not working)
2.7.3. close file descriptors/streams
- >&- 2>&- 3>&- # close 1,2,3
- <&- # close 0
2.7.4. "2>&1 >file" vs "> file 2>&1"
2>& - do nothing becouse 0,1,2 -> points to terminal
command >file 2>&1 (or just &>)
- stdout = file
- stderr = file
command 2>&1 >file - redirects only stdout to file
- stderr is redirected to stdout
- stderr = stdout_target
- stdout = file
2.7.5. stderr to pipe
command 2>&1 >/dev/null | grep 'something'
- First copy stderr to stdout - all stderr
- redirect stdout to /dev/null
- pipe stdout
2.7.6. redirect stdout to file and to stdout
command | tee >(tee) > file
2.7.7. named pipe
the FIFO file must be opened on both ends
- mkfifo pipe0 # create file
- mkfifo -m 644 pipe0
- echo "input message" > pipe0
- cat pipe0
2.7.8. not all output?
script -c 'python -i <<< "print \"test\""'
2.8. String processing
get length
${#my_string}
- echo '$(echo "upg")' = > $(echo "upg")
- echo "$(echo "upg")" = > upg
- The special parameters * and @ have special meaning when in double quotes in special cases
- a="as\nasd $BASH $(echo wtf)(new line)"
- a='as\nasd $BASH $(echo wtf)(new line)'
- echo $a => as\nasd $BASH $(echo wtf)
- echo "$a" => as\nasd $BASH $(echo wtf)(new line)
- echo -e $a => as(new line)asd $BASH $(echo wtf)
- echo -e "$a" => as(new line)asd $BASH $(echo wtf)(new line)
- printf '%q\n' "$a" => $'as\\nasd $BASH $(echo wtf)\n'
general
- cat
- sed
- grep
- awk - excellent slicer and more
- perl/python
special
- sort
- head/tail
- cut
- xargs - convert several lines to one line
- uniq
- cmp/diff
- tr replace characters
- nl number lines of files
- paste merge lines of files
- rev reverse lines characterwise
- fold
- shuf get random lines
- for item in list; do command $item done
- expand and unexpand: convert between tabs and spaces
- cut print part of each line of file separated by TAB or delimiter
- paste and join: read two files into one
- fmt, pr, fold, column, nl : text formatting
2.8.1. multiline string - HereDoc
short for "Here Document," is a type of redirection
[n]<<[-]word here-document delimiter
END, EOT, and EOF are common for world
If any part of word is quoted:
- the delimiter is the result of quote removal on word
- the lines in the here-document are not expanded, otherwise lines are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.
‘<<-’ - all leading tab characters are stripped from input lines and the line containing delimiter.
quoting:
cat << EOF $(echo Hello) EOF cat << 'EOF' $(echo Hello) EOF cat << "EOF" $(echo Hello) EOF cat << \EOF $(echo Hello) EOF
Hello $(echo Hello) $(echo Hello) $(echo Hello)
declare variable:
a=$(cat <<EOF s s EOF ) echo -e "$a"
s s
as=$(echo -e "s\ns")
echo $a # s s echo "$a" # s\ns
cat - внешняя программа которой нет в bash
set +e read -rd '' EMACS_EVAL <<EOF (progn (org-babel-tangle-file "./index.org")) EOF set -e
piping:
cat << 'EOF' | base64 -d SGVsbG8KV29ybGQK EOF
Hello World
Write to a file. The way to create script at remote server.
cat <<EOF > print.sh #!/bin/bash echo \$PWD echo $PWD EOF
function - stores the information provided by the HereDoc into variables.
readLines(){ read greeting read name } readLines << EOF Hello $USER EOF echo $greeting echo $name
Hello u
executing multiple commands on a remote machine:
ssh username@host << EOF echo hello! # this commands will be executed on remote machine echo do your work! EOF sftp username@host << EOF put test.sh # run multiple SFTP commands automatically EOF
2.8.2. sort
sort by column:
sort -t, -nk3 user.csv
- -t, - defines your delimiter as ,. (default delimeter - tab ? )
- -n - gives you numerical sort. Added since you added it in your attempt. If your user field is text only then you dont need it.
- -k3 - defines the field (key). user is the third field.
sort -k 2,2 -k 1,1 Hogwarts # sort by second and by first column
2.8.3. cut - get column
cut -d "delimiter" -f (field number) file.txt
- –complement
- –output-delimiter=”delimiter”
cut -c [(k)-(n)/(k),(n)/(n)] filename
- k denotes the starting position of the character
- n denotes the ending position of the character in each line
- if k and n are separated by “-” otherwise they are only the position of character in each line from the file taken as an input.
ps -aux | tr -s ' ' | cut -d ' ' -f 2
2.8.4. sed
Default only change the first occurrence of the word on a line.
get line:
echo -e "as\nasd" | sed -n '2p' => asd
split file by first occured character and number lines:
sed -z -e 's/\//\\:\\:\n\//' -e 's/*/\\:\\:\n\*/' | sed -e '0,/^[^\/*]/ s/^[^\/*]/\\:\\:\n/' | nl
2.8.5. grep
- filter lines by pattern -
- case insensitive grep -i "FILe"
- -o only matching
- -v invert match or NOT
- -i case insensitive
- -A num Print num lines of trailing context after each match. See also the -B and -C options.
- -B num Print num lines of leading context before each match. See also the -A and -C options.
- -C[num] Print num lines of leading and trailing context surrounding each match. The default is 2 and is equivalent to -A 2 -B 2. Note: no whitespace may be given between the option and its argument.
- -m N - stop reading file after N matching lines
OR
- dmesg | grep -i -E 'xx:xx.x|wlan|iwl|80211'
- dmesg | grep -i -e 'xx:xx.x' -e wlan -e iwl -e 80211
- grep -Ev "(127|192)"
first match
- ip a | grep -o -m 1 wlan.
search in files:
- grep -Rin patern
- n - line number, R - recursive
extract http links
- cat a.html | grep -o 'href=\".media[^\"]*/dl[0-9]*[^\"]*' | grep -o "/.*" | sed 's#^#http://site.onion#' | xatrs torsocks wget –user-agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
2.8.6. python
split paragraphs:
- cat aa | python -c "import sys; print(sys.stdin.read().split('123'))"
- ['asasd\n\n\n asdasd\n\n','asd\nas']
2.8.7. perl
split paragraphs:
bad try:
- echo "aavbb" | perl -e '@a=<>; print( (split(v, join "", @a))[0] )' => "aa"
2.8.8. awk
- echo -e "a\navbb" | awk -v RS='v' 'NR==1' - get a\na
- ps ux | awk '{if($8=="Z") print}' - get column 8 if == "Z"
- ps | awk '{print $2}' - get column 2
2.8.9. uniq
MUST BE USED AFTER sort
- -s n - skip n first characters
- -w n - sort by n first characters
- –group - separate duplicates
- -c - count unique
2.8.10. strip remove leading and trailing white spaces
vv=" asd " v=$(echo $vv) echo $v
2.8.11. tail
skip first line of output:
route -n | tail -n +2
search sentence or word in files with extension:
grep -r -i --include=\*.txt 'searchterm' ./
- -r: recursively
- -i: ignore-case
- –include: all *.txt: text files (escape with \ just in case you have a directory with asterisks in the filenames)
- 'searchterm': What to search
- ./: Start at current directory.
add prefix to every string
echo -e "a23d\nasdasd" | sed 's/^/worker: /'
2.9. Process Control
- top 2.24
- ps 2.23
- kill/killall/pkill
- free
- pgrep
- fg - background child processes
- jobs
- -l list
- -p process numbers
background
- nohup command & - even after exiting the shell
- timeout 15 command &
- ( sleep 1000 & ) - deattached from bash
Which process block file/folder?
- ls -l /proc/*/fd/* | grep ~/Downloads
- lsof/lsfd -l ~/Downloads
- fuser Downloads
$! - get pid of last started background child proces
2.10. Programming
2.10.1. shebang
#!/usr/bin/env bash #!/usr/bin/env sh
On most but not all systems, I recommend using:
#!/bin/bash #!/bin/sh
2.10.2. pros and cons
pros:
- Simple, easily understood
- Quick to write
- Doesn't take up much disk space
- Reasonable chance of working on multiple operating systems without modification, if you know what you're doing
cons:
- Slow execution (typically)
- No sane or robust access to floating point math, complex data structures, database query results, etc. (depending on language)
- Shebang line must be edited for each target system
2.10.3. Hello World.sh, scriptinga
#!/bin/sh #or #!/usr/bin/env bash set -u # Report Non-Existent Variables set -e # It terminates the execution when the error occurs. (does not work with piped commands. use Set -eo pipefail) set -o pipefail # exit execution if one of the commands in the pipe fails. # set -x # write to standard error a trace for each command # set -n # do not execute only check syntax declare -r -i PORT_NUMBER=8080 # integer, read only declare -a my_array=( apple orange ) # indexed array # Processes a file. # $1 - the name of the input file # $2 - the name of the output file main(){ local -r input_file="$1"; shift local -r output_file="$1"; shift } main "${@}"
2.10.4. arguments
- $@ - all
- $0 - script name
- $1-… - argumensts
easy to swap in future:
main(){ local -r input_file="$1"; shift local -r output_file="$1"; shift }
2.10.5. array
- array=(item1 item2 item3)
- echo ${array[*]} # echo item1 item2 item3
- set – item1 item2 item3
- echo "$3" # Arrays indices start from 1
- set – "$@" 4 # add item
- shift 2 # Remove first 2 items
- for item in "$@"; do echo "$item"; done # loop
- “${99}” - “$98” is not working
Associative Arrays / Hash Maps
- hm="$(hm_create)"
- hm_put "$hm" mykey myvalue
- hm_get "$hm" mykey
length:
- echo ${#array[@]}
Splice an Array:
- ${arrayname[@]:start:end}
from start point to end:
- v=${#a[@]}
- echo ${a[@]:1:$v}
2.10.6. comparision Posix
- ! EXPRESSION The EXPRESSION is false.
- -n STRING The length of STRING is greater than zero.
- -z STRING The lengh of STRING is zero (ie it is empty).
- STRING1 = STRING2 STRING1 is equal to STRING2
- STRING1 != STRING2 STRING1 is not equal to STRING2
- INTEGER1 -eq INTEGER2 INTEGER1 is numerically equal to INTEGER2
- INTEGER1 -gt INTEGER2 INTEGER1 is numerically greater than INTEGER2
- INTEGER1 -lt INTEGER2 INTEGER1 is numerically less than INTEGER2
- -d FILE FILE exists and is a directory.
- -e FILE FILE exists.
- -r FILE FILE exists and the read permission is granted.
- -s FILE FILE exists and it's size is greater than zero (ie. it is not empty).
- -w FILE FILE exists and the write permission is granted.
- -x FILE FILE exists and the execute permission is granted.
Boolean Operations
- && - is and
- || - is or
-a means AND and -o means OR
2.10.7. numbers comparision
must be checked agains tested not empty values
- = and != is safe for empty values "$n" = 1 - must be in ""
- num1 -eq num2 check if 1st number is equal to 2nd number
- num1 -ge num2 checks if 1st number is greater than or equal to 2nd number
- num1 -gt num2 checks if 1st number is greater than 2nd number
- num1 -le num2 checks if 1st number is less than or equal to 2nd number
- num1 -lt num2 checks if 1st number is less than 2nd number
- num1 -ne num2 checks if 1st number is not equal to 2nd number
2.10.8. strings comparision
variables must be in ""
- var1 = var2 checks if var1 is the same as string var2
- var1 != var2 checks if var1 is not the same as var2
- var1 < var2 checks if var1 is less than var2
- var1 > var2 checks if var1 is greater than var2
- -n var1 checks if var1 has a length greater than zero
- -z var1 checks if var1 has a length of zero
2.10.9. if
- [] is just a reference to the command test
- (( )) - ariphmetic
statements:
- if [ $c == 5 ]
- then echo ok
- elif [ <some test> ]
- then something
- else something
- fi
if [ "5" == 6 ] ;then echo ok;else echo bad;fi
test $c = 5;echo $?
ERROR: if[ or [$c or 5] or 5 ];
- double square brackets
- Single [] are posix shell compliant condition tests. Portable.
- Double [[]] are an extension to the standard [] and are supported by bash and other shells (e.g. zsh, ksh) For example: || instead of -o and regex matching with =~
- [ is POSIX
- [[ is a Bash extension inspired from Korn shell
Use [] whenever you want your script to be portable across shells. Use [[]] if you want conditional expressions not supported by [] and don't need to be portable.
2.10.10. loop
- while - while COMMAND; do done
- until - same as while and inverse
- for - for v in VARIABLE; do ? done ; if no "in VARIABLE" - in "$@" is assumed
- select word in words - same as echo words | while IFS=" " read word; do
while IFS= read -r line do echo "$line" done <<< "$the_list" while IFS=, read -r name familyname address; do printf 'Got "%s", "%s", and "%s"\n' "$name" "$familyname" "$address" done <file.csv while read line ; do echo $line done < <( ... code ... ) your_code | while read line; do echo $line done # while read line; # do # echo $line # done < $lines echo -e "$a" | while IFS=":" read f v; do echo "$f _ $v" # f - first column v - all others done set -- "123 3456" for v in "$@"; do $v done for j in `seq 0 24`; do # safest ... done select DRINK in tea cofee water juice appe all none do case $DRINK in tea|cofee|water|all) echo "Go to canteen" ;; juice|appe) echo "Available at home" ;; none) break ;; *) echo "ERROR: Invalid selection" ;; esac done # loop files in current directory for filename in *; do echo "put ${filename}"; done # command for every line of file - can not use variables xargs -rd'\n' -I@ -a p2.txt echo @
2.10.11. file comparision
- -d file checks if the file exists and is it’s a directory
- -e file checks if the file exists on system
- -w file checks if the file exists on system and if it is writable
- -r file checks if the file exists on system and it is readable
- -s file checks if the file exists on system and it is not empty
- -f file checks if the file exists on system and it is a file
- -O file checks if the file exists on system and if it’s is owned by the current user
- -G file checks if the file exists and the default group is the same as the current user
- -x file checks if the file exists on system and is executable
- file A -nt file B checks if file A is newer than file B
- file A -ot file B checks if file A is older than file B
2.10.12. functions
- function must precede call.
- exit - stop script
- return - stop function
function_name () { if [ -z $1 ] ;then return 1;fi # Checks if any params. echo $1 # first parameter if [ $2 ] ; then echo "Parameter #2 is $2." fi ret="val" # return value return 32 # set $? status } func2 # call with no params func2 first # call with one param b(){ echo wtf;}
2.10.13. debug
Display commands and their arguments as they are executed.
- bash -x script-name
- set -x - for bash
Display shell input lines as they are read.
- set -v - for bash
export _DEBUG=on # intelligent DEBUG function
there is two way for debug: trap command and PROMPT_COMMAND="echo Bye"
2.10.14. exit status
- $?
- exit 5
exit codes:
- 0 success
- 1-255 failure (in general)
- 1 : Miscellaneous errors, such as “divide by zero” and other impermissible operations.
- 2 : Missing keyword or command, or permission problem.
- 126 the requested command (file) can't be executed (but was found)
- 127 command (file) not found
- 128 according to ABS it's used to report an invalid argument to the exit uiltin, but I wasn't able to verify that in the source code of Bash (see code 255)
- 128 + N the shell was terminated by the signal N (also used like this by various other programs), invalid argument to exit.
- 255 wrong argument to the exit builtin (see code 128)
2.10.15. generate random
2 byte unsigned decimal integers,
$ od -vAn -N2 -tu2 < /dev/urandom 24352
1 byte signed decimal integer,
$ od -vAn -N1 -td1 < /dev/urandom -78
4 byte unsigned decimal integers,
$ od -vAn -N4 -tu4 < /dev/urandom 3394619386
seconds since epoch
date +%s%N | cut -b10-19
RANDOM env variable
echo $RANDOM % 1000 + 1 | bc
shasum
head -100 /dev/urandom | sha1sum | cut -d ' ' -f 1
2.10.16. documentation
# Processes a file. # $1 - the name of the input file # $2 - the name of the output file process_file(){ }
2.10.17. best practices
- safer scripts, see 2.11:
- set -euxo pipefail
- Instead of, use:
- cd "${foo}"; … ; cd ..
- ( cd "${foo}" … )
- pushd "${foo}"; … ; popd
- variables:
- preffer local
- global make readonle declare -r
- always refer as "${var}"
- Prepend a command with \ to override alias/builtin lookup. E.g.:
- t() { ls ;}
- alias t="ls .."
- t # refer to alias
- \t # refer to function
- For various reasons, printf is preferable to echo
- error() { printf "${red}!!! %s${reset}\\n" "${*}" 1>&2 ; }
- Name heredoc tags with what they’re part of, like:
cat <<HELPMSG usage $0 [OPTION]... [ARGUMENT]... HELPMSG # Single-quote heredocs leading tag to prevent interpolation of text between them. cat <<'MSG' [...] MSG # 7) # this won't work: sudo printf "..." > /root/some_file # this will: printf "..." | sudo tee /root/some_file > /dev/null 8) if # Don't do this if [ "$#" -ge "1" ] && [ "$1" = '-h' ] || [ "$1" = '--help' ] || [ "$1" = "-?" ]; then usage exit 0 fi # Do this help_wanted() { [ "$#" -ge "1" ] && [ "$1" = '-h' ] || [ "$1" = '--help' ] || [ "$1" = "-?" ] } if help_wanted "$@"; then usage exit 0 fi
- Check whether the script is running in the correct folder
- Allow the script to run in any folder
- Allow the script to run in any folder and locate input and output based on the environment
- Require the script to be run from a certain folder and test if folder is good.
- write log:
echo "[${now}] Logfile for $scriptName: $logFile" 2>&1 | tee $logFile
- Create documentation with -h and –help
- Clean up function
finish() { result=$? # Your cleanup code here exit ${result} } trap finish EXIT ERR # secuted on EXIT or ERR
grep pattern file | awk ‘{print $1}’
awk ‘/pattern/{print $1}’
2.10.18. export
export FLASK_RUN_HOST=localhost FLASK_RUN_PORT=8080 ; flask --app main run --no-debug
new bash session will have exports.
- -n NAME - disable export
- -f - export function
- -p - same as env but in order of history of exports.
2.10.19. null command
creates the effect of block comments in Bash scripts.
2.10.20. multiline comment
: << 'END' This is a comment END
$100
2.10.21. read
Read a line from the standard input.
- -d delim continue until the first character of DELIM is read, by default '\n'
- -r If this option is given, backslash does not act as an escape character. The backslash is considered to be part of the line.
read -ei asd: aaa ; echo $aaa >> asd:22345r >> asd:22345r
read -d '' <<EOF ... EOF
Collect all input until EOF, exit code = 1
read -d '' AAA # echo $AAA mypy -c "$AAA"
2.11. set builtin
see 2.10.13
help set
Using + rather than - causes these flags to be turned off.
The current set of flags:
echo $- # himBHs
- – Assign any remaining arguments to the positional parameters.
- - Assign any remaining arguments to the positional parameters.
- -e Exit immediately if a command exits with a non-zero status.
- -u Treat unset variables as an error when substituting.
- -f Disable file name generation (globbing). disables filename expansion also known as globbing.
- -o pipefail exit execution if one of the commands in the pipe fails.
- -v Print shell input lines as they are read.
- -x Print commands and their arguments as they are executed.
- -C If set, disallow existing regular files to be overwritten by redirection of output
- -T If set, the DEBUG and RETURN traps are inherited by shell functions.
set -e may result in some difficult to debug issues during loops because a failed command will only exit the subshell created by the pipe
2.11.1. usage
Prevent unexpected errors, make shell safer:
set -euxo pipefail
# "unofficial" bash strict mode # See: http://redsymbol.net/articles/unofficial-bash-strict-mode set -o errexit # Exit when simple command fails 'set -e' set -o errtrace # Exit on error inside any functions or subshells. set -o nounset # Trigger error when expanding unset variables 'set -u' set -o pipefail # Do not hide errors within pipes 'set -o pipefail' set -o xtrace # Display expanded command and arguments 'set -x' IFS=$'\n\t' # Split words on \n\t rather than spaces
2.11.2. traps
Defines and activates handlers to be run when the shell receives signals or other conditions.
trap --help
- -l print a list of signal names and their corresponding numbers
- -p display the trap commands associated with each SIGNAL_SPEC
Besides traps(handlers) for signals, bash have 4 special traps:
- EXIT to run on exit from the shell.
- RETURN to run each time a function or a sources script finishes.
- ERR to run each time command failure would cause the shell to exit if set -e is used.
- DEBUG to execute before every command.
trap "echo $BASH_COMMAND" DEBUG
add variable trace attribete to see when it used:
declare -t VARIABLE=value trap "echo VARIABLE is being used here." DEBUG
set -E # If set, the ERR trap is inherited by shell functions. set -o pipefail # exit execution if one of the commands in the pipe fails. set -u # Report Non-Existent Variables trap 'echo $? $LINENO :${BASH_LINENO[@]} "$BASH_COMMAND" ::${FUNCNAME[@]}' ERR
2.12. ls
directory - flat file that maps an name to inode on the filesystem.
- ls -al -tr - sort by time, reversal (latest at the bottom) (desc)
- ls * - directories and their files/folders - deep 1
- ls -d */ - only directories
- ls -d .* - dot files
- ls -d /home/user/{,.} - directories and all files, include hidden with full path
- ls -F appends symbols to filenames */=>@|
- @ means symbolic link (or that the file has extended attributes).
- * means executable.
- = means socket.
- | means named pipe.
- > means door.
- / means directory.
output in columns with classify
l='ls -F --color' ; paste -d: <($l | grep '/$') <($l | grep '*$') <($l | grep '@$') <($l | grep -v '[/@*]$') | tac| column -t -s ":" -N FOLDERS,EXECUTABLES,LINKS,OTHER
2.12.1. get direcotories
- ls -d */
- ls -l | grep ^d
- find . -maxdepth 1 -type d
- echo */
- tree -d -L 1
2.12.2. Linux File Timestamps
- mtime - modified time - write file's conetents
- ctime - change time - tracks metadata changes such as ownership and permissions. It includes renaming a file.
- atime - (access time) anyone accessed the file
ls - default mtime
- u - atime
- c - ctime
2.13. pushd/popd and dirs
- dirs -v - to print stack
- dirs - cd history without pushd/popd.
- dirs +0 - current directory
- dirs +1 - last directory
- ~1 = dirs +1
- ~-0 dirs -0 - first directory in history
https://www.howtogeek.com/659146/how-to-use-pushd-and-popd-on-linux/
2.14. sh script
#!/usr/bin/env bash
set -euxo pipefail
2.15. source & ./ & exec
in bash . and source equal in POSIX:
- ./script creates a new shell and executes each command in the script within the new shell. When the end of the script file is encountered, the new shell exits. Any changes in the new shell caused by the script are lost when the shell exits. (no screen formations)
- source execute a shell script within the context of the current shell. Since execution takes place within the context of the current shell, any changes in the shell are retained following the completion of the shell. (like by yourself)
- . script - in the current shell without forking a sub shell (equal to source)
- exec ./script - exit shell after finish
The exec command will execute a command in place of the current shell, that is, it terminates the current shell and starts a new process in its place.
exec was often used to execute the last command of a shell script. This would kill the shell slightly earlier; otherwise, the shell would wait until the last command was finished. This practice saves a process and some memory.
try exec ls. you will be logged out from your login shell.
2.16. ariphmetic
let m=(4 * 1024) echo $((2*2)) ((m-=3)) # m=m-3 ((m++)
the let command is not specified by POSIX®. The portable alternative is: [ "$(( <EXPRESSION> ))" -ne 0 ]
echo $((2#010)) - binary to decimal
expr
- is command should be preffered
- cannot assign directly to bash variables but instead returns a result on stdout
- takes each operator it recognizes as a separate word and then concatenates them into a single expression that's evaluated according to it's own rules (which differ from shell arithmetic).
2.17. execute command
$() or ``
. file == source file - execute commands in file in current shell. no need x right ./file - execute file - need x right
$( ) has the side-effect of removing trailing newlines from the results.
2.18. xargs
- -0, –null - Input items are terminated by a null character instead of by whitespace - file or directory names containing spaces or newlines are correctly handled
- transfer input to one line usable as arguments (default echo)
- xargs -n 2 # execute command for every two argument, ex diff
ex
- find /path -name '*.foo' | xargs -P 24 -I '{}' /cpu/bound/process '{}' -o '{}'.out
- -P n - Run up to max-procs processes at a time
- cat lines | xargs | tr ' ' ',' => line1,line2,line3,line4
2.19. history save with time
with timestamp echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bash_profile
2.20. id -u user
get user id
2.21. diff
https://www.math.utah.edu/docs/info/diff_3.html
- default
- “Copied” context with -c
- “Unified” context with -u Instead of displaying the two files separately, it merges them together.
diff one two
Default context - no longer widely used (used by git)
- `8a12,15' (lar) - after line 8 of one append lines 12-15 of two.
- '5,7c8,10' (fct) - change lines 5–7 of one to read as lines 8–10 of two
- `5,7d3' (rdl) - delete lines 5–7 of one (which at 3 in two)
- < - in one
- > - in two
“Copied” context - has context lines without pre-symbols
- '!' - changing like
- '+' - line of two file (nothin in one)
- '-' - line of one file (nothin in two)
“Unified” context - more compact because it omits redundant context lines. GNU patch can apply this format.
- * from-file from-file-modification-time
- — to-file to-file-modification time
- @@ from-file-range to-file-range @@
- '+' line of two file
- '-' line of one file
Differences Side by Side diff -y :
- `|' The corresponding lines differ, and they are either both complete or both incomplete.
- `<' The files differ and only the first file contains the line.
- `>' The files differ and only the second file contains the line.
- `(' Only the first file contains the line, but the difference is ignored.
- `)' Only the second file contains the line, but the difference is ignored.
- `\' The corresponding lines differ, and only the first line is incomplete.
- `/' The corresponding lines differ, and only the second line is incomplete.
2.22. date
- date +%c –date "2022-06-20" => Mon Jun 20 00:00:00 2022
- date +%F –date='@2147483647' => 2038-01-19
- date –date '06/02/2016 09:01:39' +"%s" => 1464872499
- date +%s => 1656480511
2.23. ps
https://www.computernetworkingnotes.com/linux-tutorials/ps-aux-command-and-ps-command-explained.html
a:- This option prints the running processes from all users. u:- This option shows user or owner column in output. x:- This option prints the processes those have not been executed from the terminal.
ps –eo pid,user,%cpu,cmd –-sort=-%cpu | head –n 4 ps –A –forest
notes:
- CPU usage is expressed as the percentage of time spent running during the entire lifetime of a process.
- The SIZE and RSS fields don't count some parts of a process including the page tables, kernel stack, struct thread_info, and struct task_struct.
- SIZE is the virtual size of the process (code+data+stack).
- Processes marked <defunct> are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly.
- If the length of the username is greater than the length of the display column, the username will be truncated.
PID of the bash script you can use variable $$
2.23.1. Column Description
USER The user account under which this process is running PID Process ID of this process %CPU CPU time used by this process (in percentage). %MEM Physical memory used by this process (in percentage). VSZ Virtual memory used by this process (in bytes). RSS Resident Set Size, the non-swappable physical memory used by this process (in KiB) TTY Terminal from which this process is started. Question mark (?) sign represents that this process is not started from a terminal. STAT Process state. Explained in next table. START Starting time and date of this process TIME Total CPU time used by this process COMMAND The command with all its arguments which started this process
2.23.2. stat code with description
D uninterruptible sleep (usually IO) R running or runnable (on run queue) S interruptible sleep (waiting for an event to complete) T stopped by job control signal t stopped by debugger during the tracing w paging (not valid since the 2.6.xx kernel) x dead (should never be seen) Z defunct ("zombie") process, terminated but not reaped by its parent < high-priority (not nice to other users) N low-priority (nice to other users) L has pages locked into memory (for real-time and custom IO) s is a session leader l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
- is in the foreground process group
2.24. top
include uptime command
top -d 0.5 # faster cat /proc/loadavg
sort:
- ps -o RES
- ps -o +%MEM
- f - Field manager
- s - current sort field,
- q - quit
control:
- ? - help
- parent-child - V key
- kill: press k and enter PID
- bold/colur - b/z
- renice - r
- </> - sort by column
memory:
- %MEM: The percentage of the system’s physical memory in use by this process
- VIRT: Virtual memory size
- RES: Resident set size: the total physical memory in use by this process
sort by VIRT memory
for i in {1..10};do date; top -b -o +VIRT | head -n 17|tail -11;sleep 5;done
2.24.1. view description
first row:
- current time (22 :14:26)
- uptime of the machine (up 155 day, 16:33)
- users sessions logged in (2 users)
- average load: 1, 5, 15 - exponentially-damped moving sums of a five second average
second row:
- Total Processes running (866 total)
- Running Processes (8 running)
- Sleeping Processes (856 sleeping)
- Stopped Processes (0 stopped)
CPU line: sum up all the percentages the total will be 100% of the cpu
- us, user : time running un-niced user processes in percentage(18.6%us)
- sy, system : time running kernel processes
- ni, nice : time running niced user processes
- id, idle : time spent in the kernel idle handler
- wa, IO-wait : time waiting for I/O completion
- hi : time spent servicing hardware interrupts
- si : time spent servicing software interrupts
- st : time stolen from this vm by the hypervisor
processes
- PID – ID of the process(819328)
- USER – The user that is the owner of the process (root)
- PR – priority of the process (20)
- NI – The “NICE” value of the process (0)
- VIRT – virtual memory used by the process (3423m)
- RES – physical memory used from the process (3.3g)
- SHR – shared memory of the process (1788)
- S – indicates the status of the process: S=sleep R=running Z=zombie (S)
- %CPU – This is the percentage of CPU used by this process (65.9)
- %MEM – This is the percentage of RAM used by the process (5.2)
- TIME+ –This is the total time of activity of this process (1363:48)
- COMMAND – And this is the name of the process (exim)
2.25. pgrep
get all processed by name and kill them
pgrep python | xargs kill -s 9
2.26. vmstat
https://www.howtogeek.com/425232/how-to-securely-delete-files-on-linux/
Since last reboot
vmstat 2 -d -S M
- 2 sec recreate
- -a - active and inactive memory
- -f - shows the number of tasks that have been launched
- -d - disk
- -D - summary statistics for your disk activity
- -p sda1
2.27. pmap
report memory map of a process
pmap -x $(pidof bash | cut -d ' ' -f 1) | head -n-2 | tail -n+3 | grep -v deleted | sort -k2 -n | awk '{sum+=$2} END {print sum/1024 " MiB"}'
2.28. find
find files by name and execute command:
- find . -type f -name "*.txt" -exec ls -l {} +
- find . -type f -name "*.txt" | xargs ls -l
- find . -type f -name "*.txt" -exec mv {} {}.old \;
multiple file extensions:
- find . -type f \( -name "*.sh" -o -name "*.txt" \)
find directory:
- find . -type d -iname So* -ls
- -ls - print in ls -al format
2.28.1. find [options] [starting-point…] [ex‐pression]
expressions:
- TESTS:
- -amin n File was last accessed less than, more than or exactly n minutes ago.
- -empty File is empty and is either a regular file or a directory.
- -executable Matches files which are executable
- -group gname File belongs to group gname (numeric group ID allowed).
- -iname pattern Like -name, but the match is case insensitive.
- -name pattern Base of file name
- -links n has less than, more than or exactly n hard links.
- -nouser No user corresponds to file's numeric user ID.
- -perm -a+r,u+w
- -readable Matches files which are readable by the current user.
- -size 2M
- -type c
- b block (buffered) special
- c character (unbuffered) special
- d directory
- p named pipe (FIFO)
- f regular file
- l symbolic link;
- s socket
- D door (Solaris)
- -user uname File is owned by user uname (numeric user ID allowed).
- ACTION
- -delete
- exec command ;
- -print full file name
- -quit exit
when {} used - at the end must be:
- + - result: ls -l file1 file2 file3
- \; - {} may be used more than once in the same exec statement, result: ls -l file1, ls -l file2
2.29. dd
2.29.1. usecases
create: dd if=/dev/zero of=test.img bs=1 count 0 seek=10G
resize +2G: e2fsck -f test.img resize2fs test.img 12G e2fsck -f test.img
— Iso Disk write: cdrecord dev=0,0,0 -eject -dao debian-testing-amd64-xfce-CD-1.iso
dd if=/a.iso of=/dev/sdc bs=8192k
dd if=tails.img of=/dev/sda bs=16M oflag=direct status=progress
dd if=gentoo.iso of=/dev/sda bs=4M oflag=direct status=progress
dd bs=16M if=/dev/sda3 | ssh root@serverB "dd bs=16M of=/dev/sda3"
2.29.2. bs
read and write up to BYTES bytes at a time (default: 512); overrides ibs and obs
- block size theory
Block devices and block sizes
- For example hard disks, cdrom drives and floppies are block devices, but not the keyboard.
- file system needs to split up a partition into blocks to store files and file parts (another block size)
- Most hard drives shipped with 512 byte sector/block sizes.
Logical and Physical Sector Size
- Advanced Format, same as 4K sector technology, uses the storage surface media more efficiently by combining
data that would have been stored in eight 512-byte sectors into one single sector that is 4096 bytes (4 KiB) in length.
- any data transfers between the disk and the computer still use the common 512-byte block size.
- the disk is addressable in 512-byte blocks, but performance will be better if the OS arranges the disk operations to take into account the 4096-byte internal block size.
filesystem block size:
- tune2fs -l /dev/sdb2
- file -s /dev/sda1
- stat -fc %s .
- blockdev –getbsz /dev/sda # change when mounted
Kernel block size
- All linux blocks are currently 1024 bytes.
2.29.3. cteate disk
dd if=/dev/zero of=/mnt/sda5/tdisk.img bs=1 seek=40G count=1 Discarding device blocks: 4096/10485760 done Creating filesystem with 10485760 4k blocks and 2621440 inodes Filesystem UUID: 351a17dd-246d-49f5-b871-dde0cf38c1c4 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624
Allocating group tables: 0/320 done Writing inode tables: 0/320 done Creating journal (65536 blocks): done Writing superblocks and filesystem accounting information: 0/320 done
mkfs.ext4 tdisk.img
.
2.30. access control to files
- lsattr
- chattr
- chmod
- chown
- stat
2.31. for external storages
- lsblk -p - list block devices, find connected USB Flash
- parted - partition table
- parted -l - list partitions on all disks
- fdisk - partition table
- fdisk -l
- dd
- status=progress - useful to see what's happening
- sync - Synchronize cached writes to persistent storagem, when u write to USB
2.32. filesystem block size
- tune2fs -l /dev/sdb2
- file -s /dev/sda1
- stat -fc %s .
- blockdev –getbsz /dev/sda # change when mounted
2.33. tar
tar.gz:
- crate: tar -czf a.tar.gz /
- extract: tar xpzf –xattrs-include=' .' –numeric-owner a.tar.gz
- -p, –preserve-permissions (default for supe ruser)
- inspect: tar ztf/tf m.tar.gz/m.tar
- create from files: tar –totals -cvf a.tar file1 file2 file3
2.34. compression
TOP for texts:
- 7z - LZMA compression algorithm
- xz
- bzip2 - BWT
- rar
- gzip - LZ77
- zip
TOP for mixed:
- rar
- 7z
- zip
- bzip2
- gzip
my for texts 518MB :
- time tar.xz - real 1m49.582s, user 1m50.062s - 44M
- time tar.7z - real 0m55.497s, user 1m43.210s - 44M
- time tar.bz2 - real 0m33.160s, user 0m31.348s - 48M
- time tar.gz - real 0m10.430s, user 0m8.876s - 59M
texts < 14370 chars:
cat file | gzip -f - |base64 | base64 -d | gzip -d -
for text > 14370 chars:
cat file | bzip2 |base64 | base64 -d | bzip2 -d
2.34.1. tar.7z
compress:
tar cf - <source folder> | 7z a -si <Destination archive>.tar.7z
decompress
7z x -so yourfile.tar.7z | tar xf - -C target_dir
2.34.2. zip
create:
- zip a f # creat a.zip with f file
- zip -r a f # creat a.zip with with f folder and all it's files
inspect archive: unzip -l a.zip
extract:
- unzip test_archive.zip -d ./Music
- zip chunks/multi-part/split
to chunks:
- zip -r -s <size in megabytes>M <new_file>.zip <folder>/*
to single archive (myfiles.z01, myfiles.z02, myfiles.z03):
- zip -F myfiles.zip –out single-archive.zip
decompress:
- All parts must be concatenated together in order, and then ``zip -F'' (for zip 2.x) or ``zip -FF'' (for zip 3.x) must be performed on the concatenated archive in order to ``fix'' it
- unzip -l - list archive files
2.35. curl
- proxy support
- user authentication
- ftp upload
- SSL connections
- cookies
- file transfer resume and more
2.35.1. SOCKS5 proxy
- curl -x socks5h://localhost:8080 api.ipify.org
2.35.2. mimic browser request:
- curl –http0.9 -vvv -H "Accept: application/xml" -A "Mozilla" http://a.ru
2.35.3. POST GET
POST file:
- curl -X POST -F file1=@/art.jpg http://site.com/api/upload
- curl -i -X POST host:port/post-file -H "Content-Type: text/xml" –data-binary "@path/to/file"
GET
- curl -X GET "http://anonchek.ru/get?id=a43219ce4a4a490d9cd34beb718cb09f" -H "accept: application/json"
- curl -Gv 'https://geocode-maps.yandex.ru/1.x/?apikey=ba47e54a-61a5-49af-9c84-558d92b35a33&format=json' –data-urlencode 'geocode=Тверская 6'
you cannot really use the -d/–data option if this is not a POST request. But this is still possible if you use the -G option.
- curl -X DELETE -G 'http://localhost:5000/locations' -d 'id=3'
2.35.4. POST paramters
–url-query
- –url-query name=val
- –url-query "+name=%20foo"
- –url-query @fileonly
- –url-query name@file
- –url-query =encodethis
-d
- -d "name=curl" -d "tool=cmdline"
- -d @filename
–data-urlencode - as -d buth with urlencode
- –data-urlencode name=val
- –data-urlencode =encodethis
- -data-urlencode name@file
- –data-urlencode @fileonly
–data-raw - without the special interpretation of the @ character
–data-binary @filename
–data-ascii alias to -d
-I, –head
-F or –form
-T, –upload-file
2.36. du - file space usage
du -h -x -s – * | sort -r -h | head -20 - biggest files
2.37. man, info, help
2.37.1. history
Man pages - UNIX traditional way of distributing documentation about programs.
GNU advocates against the use of man pages at all and contends that complex software systems should have complete and comprehensive documentation rather than just a set of short man pages.
help is a built-in command in the bash shell (and that shell only) that documents some of the builtin commands and keywords of that shell
2.37.2. man
- terms
convention to the layout of man pages (sections of page): Name, Synopsis, Configuration, Description, Options, … etc.
page - name of program or function.
pages located in sections 1-9. (groups of pages) see: $ man man
- man man.7 or man 7 man or man 'man(7)'
- arguments
- -a display all pages, wihtout will exit after displaying the most suitable manual page it finds.
- -aw print list of exisitng pages
- –wildcard gl* - to use wildcard in term
- –regex gl.*
- -k search -s 3 list of all man pages which relate to 'search' in section 3
- man -P 'less -p ^EXAMPLES' man to get EXAMPLES section for man comman
- man tcpdump | less -Ip examples
- keys
- h help
- / and ? search forward/backward
- n/N search forward, backward
- Ctrl + k/n up/down
- u / d Scroll up/down by half a page
- b / space Scroll up/down by a page
- highlighting on and off, press Esc+U.
2.37.3. info
- info command line arguments
- -a
- get list of all manuals for term
- -k
- look up string in all indices of all manuals
- -d
- -O
- command line options for command
- terms:
- node - page
- keys:
navigation
- n/p or
[/]
next/previous node - u up node (level)
- l/r go back/forward - return from reference
- t top node
- d directory node - internal mode
help
- h help
- H Basic info command keys
- q quit
- C-n/p next/previous line
- //? search
- }/{ or C-x n/p next/previous
- R toggle regex search
- M-/ tree search in current node tree, Global search!
- TAB/S-TAB next/previous reference
- C-x C-b list of visiter references with locations of files
- t/T default - major nodes and detail or full one tree.
- C-x 1 Delete all other windows.
- x Delete current window.
Backspace or ] - back and up level space or [ - forward and down
b - top of buffer 1-9 - pick node in menu
< - top page > - bottom page
l - go back at history r - go forward at history
i - search for topoc and go to the index
Cross reference - link to another topic Follow cross reference - just RET or f name_of_reference f? - get list of references Tab - scroll references at page (very useful) l - go back
d - directory
- n/p or
2.38. less
man tcpdump | less -Ip examples
- I - IGNORE-CASE
- p - pattern
2.39. date
set time:
- date +%Y%m%d -s "20220429"
- date -s "19:35"
epoch to time:
- date -d @1234567890
2.40. tee
append line to file and to STDOUT
echo "as" | tee -a /etc/fstab
2.40.1. freeze problem
tee inside the process substitution will not exit until it gets an eof on its stdin or some error happens.
2.41. time
- real - time from start to finish of the call.
- user - amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only actual CPU
time used in executing the process.
- Sys - is the amount of CPU time spent in the kernel within the process.
other explanation.
- Real time is wall clock time. (what we could measure with a stopwatch)
- User time is the amount of time spend in user-mode within the process
- Sys is the CPU time spend in the kernel within the process.
Wall time, or the amount of time that passes between you starting the process and the process ending.
(Wall time) - (CPU time) = (time the CPU spent doing other things)
cases
- real < user: The process is CPU bound and takes advantage of parallel execution on multiple cores/CPUs.
- real ≈ user: The process is CPU bound and takes no advantage of parallel exeuction.
- real > user: The process is I/O bound. Execution on multiple cores would be of little to no advantage.
2.42. lsfd
files opende by processes of user: lsfd -Q 'USER == "larry"'
2.43. fuser
find process on port:
fuser -nv tcp 80
2.44. administration commands
- мониторинга процессов (ps, top, htop, atop, lsof)
- проверок производительности системы (nmon, iostat, sar, vmstat)
- утилитами диагностики сетевых подключений (ping, traceroute, mtr, nmap, netstat, tcpdupm, dig, scp),
- файрволов Linux (ufw/firewalld, iptables/nftables)
2.44.1. memory
- dmesg | grep Memory: - RAM memory available in kilobytes.
- cat /proc/meminfo, free
- free - idle memory
- buff/cache - memory used for cache but available.
- vmstat -w - first report produced gives averages since the last reboot, additional for delay period.
- vmstat -wa - active/inactive
Context switching
- Context switching can be due to multitasking, Interrupt handling , user & kernel mode switching. The interrupt rate will naturally go high, if there is higher network traffic, or higher disk traffic. Also it is dependent on the application which every now and then invoking system calls.
- If the cores/CPU's are not sufficient to handle load of threads created by application will also result in context switching.
2.45. USECASES
- print column:
- ps | awk '{print $2}'
- ps | tr -s ' ' | cut -d ' ' -f 3
- find duplicated files:
- find ./ -type f -exec md5sum '{}' ';' > dupl
- cat dupl | sort | uniq –all-repeated=separate -w 15 | grep '\.\/' | uniq -w 15 | cut -c 35- | xargs rm
- copy files from stdinput | xargs -I{} cp -u {} /destination/path
- rename multiple files: find . -depth -name "*.html" -exec sh -c 'f="{}"; mv – "$f" "${f%.html}.php"' \;
- sync directories: # rsync -aqxP /var/* /mnt/newvar
- вывод в файл и stdout: cmd | tee file
- list directories: ls -d */
- read file line by line: while read -r line;do echo "$line";done < file
- message window: xmessage -timeout 50 "MESSAGE HERE!!!"
- execute for every file: find . -name "*.jpg" -maxdepth 1 -type f -exec ~/firef \{\} \; -exec sleep 1 \; -exec pkill firefox \;
- find . -name "*.mpg" -maxdepth 1 -type f -exec sh -c "firefox {} & sleep 10 ; pkill firefox ;" \;
- for f in *.txt; do echo "$f"; done
- zip
- for f in ./* ; do zip -r "${f:2}".zip "${f:2}" ; done # all folders
- for f in ./* ; do zip -r "${f:2}".zip "$f" ; done # all files
time elapsed
SECONDS=0 ; while true ; sleep 1 ; do echo -ne "\r" $SECONDS ; done
get parent process
- ps -o ppid= -p 21509
- pstree -s -p 21509
filename filepath
- basename $fspec # get filename
- dirname $fspec # get filedir
split and rejoin big file
- split -d -b 1M file file_prefix_
- -b 1M put SIZE bytes per output file
- -d use numeric suffixes starting at 0, not alphabetic
- cat file_prefix_* > file
logs from several host to one file
- node:
script -c 'python param.py' |& sed -u "s/^/${who}: /" |& perl -ne 'use IO::Handle ; printf "%s %s", scalar time(), $_ ; STDOUT->autoflush(1) ;' |& tee -a ${who}.log
- server:
watch "cat chief_0.log worker_0.log worker_1.log | sort -ns -k1 | tail"
add string to every line in pipeline:
- <command> | sed 's/^/<string>/'
- <command> | awk '{print "<string>" $0}'
- <command> | while read line; do echo "<string>$line"; done
overwrite output
echo -en "aaaa\r" ; sleep 1 ; echo -ne "bbb\033[0K\r" ; sleep 1; echo "[INFO] Processing file: log.txt"
telega most frequient spammers
- cat messages.org | grep -o "@[^$ ]*" | grep -v '>' | sort | uniq -c | sort -r
replace substring in files
for f in ./*.el ; do cat $f | sed 's/2023/2024/g' > /tmp/t mv /tmp/t $f done
split long lines, broke line with limit
echo 'asdasd' | fold -s -w 80
2.46. links
- https://learnbyexample.gitbooks.io/linux-command-line/content/Shell.html
- Shell & Utilities https://pubs.opengroup.org/onlinepubs/9699919799/utilities/contents.html
- https://github.com/dylanaraps/pure-bash-bible
extract IP:PORT from file:
- grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}:[0-9]{1,5}" p.txt
3. Bash new features
bash –posix or set -o posix - to execute in POSIX mode
Syntax
- case
- ;;& ;& etc
- (no term)
- for ((i=0; i<3; i++)); do ; printf '%s\n' "$i" ; done
- extended glob
- +( ) @( ) !( ) *( )
- select
- construction to create menu with input from user.
- file slurp
- $(< file), in shell: $(cat file)
Expansions
- {a,b,c} or {1..10}
- <( ) and >( ) process substitutions. instead of foo <(bar), write mkfifo /tmp/foo_fifo; bar > /tmp/foo_fifo & foo /tmp/foo_fifo
Parameter Expansions
- ${name:n:l} - $(expr "x$name" : "x.\{0,$n\}\(.\{0,$l\}\)")
- ${name/foo/bar} – you can use $(printf '%s\n' "$name" | sed 's/foo/bar/')
- ${!name}
- #, ##, %, and %% - operators when used together with the @ or * parameters
Arrays, in shell: IFS and set -f
Conditionals
- [[ - use [ and use double quotes around the expansions [ "$var" = "" ]
- [ foo = *glov ]] - use case or expr or grep
- == - use = instead - only = is defined by POSIX
- compare lexicographically. < > - not POSIX
- compare modification times [ file1 -nt/-ot file2 ]] - [ "$(find 'file1' -prune -newer 'file2')" ] or [ "file1" -nt "file2" ]
- same hardlink [ file1 -ef file2 ]] - [ "file1" -ef "file2" ]
- (( )) (without the $) acts like a command on its own
Arithmetic
- ++ –
- , - comma as command separator
- **
- let or ((…)) - use [ "$((…))" -ne 0 ]
4. tmux
emerge –ask app-misc/tmux
4.1. status
[10] 0: bash*
- 10 - session number
- bash - window name, default is dynamic from current command name
4.2. terms
- server
- Servers are terminated when the last remaining running session is closed.
- session
- default session with number 0. When we start a new instance of tmux, new session is created inside the running server.
- window
- pane
- splits of window, pseudoterminals. Window always have at least one pane.
target-session (-t)
4.3. keys
Ctrl-b is the begining of commands:
- ? help
- [ now Page Up, Page down
- CTRL+SPACE to start copying
- ALT+w or CTRL+w to copy into Tmux buffer.
- ] to paste
quit:
- x kill the current panel and quit if one left in window
- & kill window and quit
- d disconnect from panel
toggle between:
- (/) toggle between sessions
- s list sessions, x - kill session, C-p/n - up/down
- w list sessions and windows
- % create a horizontal pane
- " create a vertical pane
- o toggle between panes - current surround with green lines
- array toggle between panes
- L
rename:
- $ session
- , window
- https://danielmiessler.com/study/tmux/
4.4. commands
- tmux ls - list tmux sessions
- tmux attach -t 0 - attach to session
- tmux
4.5. xorg start and terminal unused
To start terminal with tmux:
xfce4-terminal -e tmux
To kill unused terminal windows with idling single bash:
alias cl="ps aux | grep '\-bash' | grep Ss+ | tr -s ' ' | cut -f 2 -d ' ' | xargs kill -s 9"
to kill deattached tmux sessions, add to ~/.bashrc,
test -n "$DISPLAY" && tmux list-sessions | grep -v attached | cut -d: -f1 | xargs -r -n1 tmux kill-session -t &>/dev/null
4.6. xorg system clipboard - How to copy to clipboard?
Soltuion: create file ~/tmux.conf and add this line:
set -s copy-command "xclip -sel clip -i"
this app should be installed:
emerge x11-misc/xclip
4.7. kill deattached sessions
tmux list-sessions | grep -v attached | cut -d: -f1 | xargs -r -n1 tmux kill-session -t &>/dev/null
4.8. option
global options:
tmux show-options -g
For window options:
tmux show-options -w
For server options:
tmux show-options -s
4.9. rebind keys
tmux list-keys | less
bind-key -T copy-mode C-k send-keys -X cursor-up bind-key -T copy-mode C-l send-keys -X cursor-left
5. SSH
Secure SHell - port 22.
- Mozilla recommendations https://infosec.mozilla.org/guidelines/openssh
5.1. config files
- /etc/ssh/sshd_config - server conf
- /etc/conf.d/sshd
- etc/ssh/known_hosts and ~.ssh/known_hosts - list of public keys for all the hosts to which the user has connected with ssh. for verifying the identity of client's systems. protecting against impersonation or man-in-the-middle attacks.
- ~/.ssh/authorized_keys - mandatory, not added automatically
- /etc/ssh/ssh_config - client conf
- ~/.ssh/config
5.2. permissions
- 0600 ~/.ssh/authorized_keys # rw
- 0600 ~/.ssh/known_hosts
- 0644 ~/.ssh/config # rw-r-r
- 0600 ~/.ssh/id_rsa # rw
- 0644 ~/.ssh/id_rsa.pub # rw-r-r
- 0700 or 755 /ssh
- 644 /etc/ssh_config
- 600 /etc/ss_config
5.3. authentication types
- password
- PasswordAuthentication yes
- public key
- Host-based - public key + ip white list
- EnableSSHKeySign yes
- HostbasedAuthentication yes
- IgnoreRhosts no
- keyboard-interactive - server sends prompts to the Client who should provide the correct response.
- KbdInteractiveAuthentication yes
- ChallengeResponseAuthentication yes
- challenge-response - A specific backend system sends the challenges to the client’s PC and verifies the response.
- ChallengeResponseAuthentication yes
- KbdInteractiveAuthentication yes
- GSSAPI - Kerberos authentication
- GssapiAuthentication
- GssapiKeyExchange
5.4. usage
- /usr/sbin/sshd -t # validate its configuration
- ssh-add -l # list keys
- copy files
- from remote to local
- rsync username@ip_address:/home/username/filename .
- scp filename username@ip_address:/home/username .
- rsync -r source_folder/ dest_folder/ # recurse copy contents of the source folder into dest_folder
- rsync -vP username@host:/path/to/file . # download large file with progress
- from local to the remote
- rsync username@ip_address:/home/username/filename .
- scp filename username@ip_address:/home/username
- from remote to local
- ssh -t command # execute command right after login
5.5. commands
- scp - secure file copy
- sftp - secure file transfer
- ssh-add - add private key identities to the authentication agent
- ssh-agent - authentication agent
- ssh-copy-id - use locally available keys to authorize logins on a remote machine
- ssh-keygen - authentication key utility
- ssh-keyscan - gather SSH public keys from servers
- sshd - OpenSSH daemon
5.6. 1) ssh server - password
- net-misc/openssh
- Config server:
- PasswordAuthentication yes
- PermitRootLogin yes
- PubkeyAuthentication yes # allow migrate to pubkey (not required)
- ListenAddress 0.0.0.0 # and add rc_need="net.enp" to /etc/conf.d/sshd (not required)
- sshd -t
client: ssh user@127.16.0.1
5.7. 2) ssh server - generate key - public key and private key
Just don’t use ECDSA/DSA!
- create and copy key at client
- ssh-keygen -b 4096 -t rsa -f ~/.ssh/mykey_rsa
- ssh-keygen -t ed25519 -a 256 (-f ~/.ssh/id_ed25519 -C "username@hostname")
- add -q -N "" for empty passphrase
- ssh-copy-id -i ~/.ssh/id_rsa.pub 172.16.0.1
- ssh user@172.16.0.1 # try loggin
- eval `ssh-agent`
- ssh-add home/user.ssh/id_rsa
- ssh user@172.16.0.1 # try loggin
- server - change config of sshd
- PermitRootLogin no # if not needed
- PasswordAuthentication no
- HostbasedAuthentication no
- KbdInteractiveAuthentication no # keyboard-interactive
- # UsePAM yes # should be commented
- AllowUsers user1 user2 # restrict access for this users only
- sshd -t
- client - samplify ~/.ssh/config
- Host ssh-ex
- HostName 104.236.90.57
- User fideloper
- IdentitiesOnly yes # where ssh-agent offers many different identities. (optional)
- IdentityFile ~/.ssh/id_sshex
- PreferredAuthentications publickey # (optional)
- Host ssh-ex
5.8. 3) sftp server
- uncomment /etc/ssh/sshd_config: Subsystem sftp /usr…
- for autocompletion at client: echo "net-misc/openssh libedit" >> /etc/portage/package.use/openssh
5.9. 4) X11 Forwarding
to start a graphical applications installed on a remote Linux system and forward that application windows (screen) to the local system
- /etc/sshd_config
- ForwardX11=yes
- ssh -Y host
ssh -X - is not working:
- Warning: untrusted X11 forwarding setup failed: xauth key data not generated
- xset: ubable to open display ""
5.10. 5) sshfs - client
- File systems —> [*] FUSE (Filesystem in Userspace) support
- emerge –ask net-fs/sshfs
- umount: fusermount -u /path/to/local/sshfs/mount
Options for access:
- -o allow_other
- replace allow_other with allow_root, for root).
- -o umask=M
- set file permissions (octal)
- -o uid=N
- set file owner
- -o gid=N
- set file group
It is recommended to run SSHFS as regular user (not as root).
There is your options:
- run as root
- only root can access folder
- with allow_other - you must hide folder as subfolder and set permissions on parent folder
- chmod o-rwx /mnt
- chown root:users /mnt
- mkdir /mnt/share
- sshfs -o allow_others -o uid=1000 -o gid=100 target:/mnt/share /mnt/share
- run as user
- only user can access folder
- root and user can access folder but user can allow access to anyone
- with /etc/fuse.conf uncomment: user_allow_other
- sudo -u user sshfs -o allow_root -o uid=1000 -o gid=100 target:/mnt/share /mnt/share
/etc/fstab:
- as root (allow_other):
- sshfs#SERVER_USER@remotehost:/SOURCEDIR home/USER/Music fuse user,_netdev,idmap=user,transform_symlinks,identityfile=/home/USER.ssh/id_rsa,allow_other,default_permissions,uid=1000,gid=1000 0 0
- user@host:/remote/path local/path fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/user.ssh/id_rsa,allow_other,reconnect 0 0
- noauto tells it not to mount at boot
- x-systemd.automount does the on-demand magic
- _netdev tells it that it is a network device, not a block device (without it "No such device" errors might happen)
- as user:
- user@host:/remote/path local/path fuse.sshfs noauto,x-systemd.automount,_netdev,user,idmap=user,follow_symlinks,identityfile=/home/user.ssh/id_rsa,allow_other,default_permissions,uid=USER_ID_N,gid=USER_GID_N 0 0
up:
- sshfs -o allow_other -o uid=1000 -o gid=100 target:/mnt/share /mnt/share
down:
- chmod -R g+rwx /mnt/share/* # allow at target to write
- fusermount -u /mnt/share # umount
5.11. TODO Internet over socks
Viriant 1 - tun - man ssh
- https://www.copahost.com/blog/openvpn-in-ubuntu-tun-tap/
- # cat /dev/net/tun
- cat: /dev/net/tun: File descriptor in bad state
- If the output shows “File descriptor in bad state”, then it means it is working correctly.
Variant 2 - proxychains
- ssh -fNTD 127.0.0.1:<local port> username@sshserver
- -f go backgroound
- -N Do not execute a remote command. This is useful for just for warding ports.
- -T Disable pseudo-terminal allocation.
- edit /etc/proxychains.conf :
- socks5 127.0.0.1 <localport>
- proxy_dns
- proxychains <program name>
Variant 2 - tsocks
5.12. DNS over socks
- net-proxy/dnsproxy
- https://github.com/awaw/dnsproxy
- test dnsproxy: dig @127.0.0.1 -p 53000 google.com (dnsutils)
problems: resolv.conf require 53 UDP port
- you need to root to redirect to 53 port
- UDP is not forwarded with SSH directly
solution:
- sudo -u ssh ssh -L 6667:0.0.0.0:6667 -C -o ProxyCommand='nc -X 5 -x 127.0.0.1:1082 %h %p' -p 9067 root@remove_ssh
- remote:
- mkfifo /tmp/fifo
- nc -l -p 6667 < /tmp/fifo | nc -u localhost 53000 > /tmp/fifo
- local:
- socat udp-listen:53,reuseaddr,fork TCP:127.0.0.1:6667 &
- (other) # mkfifo /tmp/fifo ; sudo nc -l -u -p 53 < /tmp/fifo | nc localhost 6667 > /tmp/fifo
server:
- ssh -L 6667:0.0.0:6667 root@remove_ssh (sudo -u ssh ssh -L 6667:0.0.0.0:6667 -C -o ProxyCommand='nc -X 5
-x 127.0.0.1:1082 %h %p' -p 22 root@remove_ssh)
- socat tcp4-listen:6667,reuseaddr,fork UDP:127.0.0.1:53000
- local:
- socat udp-listen:53,reuseaddr,fork TCP:127.0.0.1:6667 &
- (bad)# mkfifo /tmp/fifo ; sudo nc -l -u -p 53 < /tmp/fifo | nc localhost 6667 > /tmp/fifo
- /etc/resolv.conf: nameserver 127.0.0.1
- prevent dhcpd overweite /etc/resolv.conf: chattr +i /etc/resolv.conf
/tmp/fifo
https://superuser.com/questions/53103/udp-traffic-through-ssh-tunnel
5.13. ssh-agent
- key manager for SSH. It holds your keys and certificates in memory, unencrypted, and ready for use by ssh
- runs in the background on your system
do not type password all the time
- eval $(ssh-agent) or eval `ssh-agent` # to start agent
- ssh-agent -k # to kill agent
- ssh-add -l # list private keys currently accessible to the agent ~/.ssh/id_rsa
- ssh-add key
- ssh-add -D - del -l list
ssh-add without any parameters - it will scan your home directory for some standard keys and add them to your agent
ssh-agent -k # kill daemon
5.14. agent forwarding problem
security risk: anyone with root access on the remote host can discreetly access your local SSH agent through the socket. They can use your keys to impersonate you on other machines on the network.
How to reduce your risk when agent forwarding:
- ssh -A turns on agent forwarding for a single session.
- ssh-add -x locks the agent with a password, and ssh-add -X unlocks it
- alternative SSH agent that prompts you when it’s being used.
- ProxyJump directive: ssh -J bastion.example.com cloud.computer.internal # to connect to
cloud.computer.internal via your bastion.example.com
- your SSH client uses keys from your agent to connect to bastion.example.com.
- Once connected, SSHD on the bastion connects to cloud.computer.internal and hands that connection off to your local SSH client.
- Your local SSH client runs through the handshake again, this time with cloud.computer.internal.
- Older versions of SSH and SSHD (prior to 7.2, released in 2016) don’t support ProxyJump:
- ssh -o ProxyCommand="ssh bastion.example.com nc %h %p" cloud.computer.internal
5.15. ssh-copy-id
ssh-copy-id user@hostname.example.com -p <port-number>
copies the public key of your default identity (use -i identity_file for other identities) to the remote host.
5.16. inside of ssh
- The client presents a public key to the server.
- The server generates and sends a brief, random message, asking the client to sign it using the private key.
- The client asks the SSH agent to sign the message and forwards the result back to the server.
- The server checks the signature using the client’s public key.
- The server now has proof that the client is in possession of their private key.
- Later in the handshake process, a set of new, ephemeral and symmetric keys are generated and used to encrypt the SSH session traffic. These keys may not even last the entire session; a “rekey” event happens at regular intervals.
5.17. old
В начале - настраиваем сервар на аутентификация по ключу
- на клиенте (без пароля): ssh-keygen -o -b 4096 -t rsa -f ~/.ssh/mykey_rsa
- на сервере по рутом /etc/ssh/sshd_config:
- PermitRootLogin yes
- PubkeyAuthentication no
- PasswordAuthentication yes
На клиенте - копируем публичный ключ на сервер в authorized_keys:
- ip = 10.0.0.1 - сервера
- scp user@$ip:"~/.ssh/mykey_rsa.pub" ~/.ssh/authorized_keys
потом на сервере :
- PubkeyAuthentication yes
- AuthorizedKeysFile .ssh/authorized_keys
- PasswordAuthentication no
- UsePAM no - Also verify UsePAM yes is not in the configuration file; it overrides the public key authentication mechanism. Alternatively PasswordAuthentication or ChallengeResponseAuthentication directives can be disabled
X: конектится по ssh сразу под тем пользователем под которым хочешь запустить X-приложение
- /etc/ssh/sshd_config
- X11Forwarding yes
- X11DisplayOffset 10
wine msiexec /i E82.40_CheckPointVPN.msi
SOCKS 5 прокси ssh -D 8888 user@33.224.56.34
restict one command with argument
- command="./nuke $SSH_ORIGINAL_COMMAND",no-port-forwarding,no-x11-forwarding,no-agent-forwarding KEY_TYPE KEY COMMENT
- ./nuke:
- set -euf -o pipefail
- echo "$@"
copy public key to remote machine
- ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.1
5.18. githib ~/.ssh/config
IdentityFile - replace for "ssh -i"
# alias: Host github.com HostName github.com Port 22 User git_username IdentityFile ~/.ssh/rsa_id PreferredAuthentications publickey
5.19. key type
- RSA - difficulty of factoring large numbers
- DSA - no longer recommended.
- ECDSA - weak - elliptic curves. Only three key sizes are supported: 256, 384, and 521 (sic!) bits.
ed25519 (EdDS) - new
https://stribika.github.io/2015/01/04/secure-secure-shell.html
5.20. security
- Protocol 2 #Protocol 1 is fundamentally broken
- StrictModes yes #Protects from misconfiguration
Diffie-Hellman key exchange
- use 2048-bit or larger primes
RFC4419: The use of multiple moduli inhibits a determined attacker from precalculating moduli exchange values, and discourages dedication of resources for analysis of any particular modulus.
generate new non elliptic-curve primes do this:
- ssh-keygen -G moduli-2048.candidates -b 2048
- ssh-keygen -T moduli-2048 -f moduli-2048.candidates
- cp moduli-2048 /etc/ssh/moduli
soft
- sshguard
- fail2ban
- Logjam Attack https://weakdh.org/
Stronger sshd_config:
- ClientAliveInterval 60 # seconds
5.21. Port Forwarding/Tunneling
https://www.techrepublic.com/article/how-to-use-local-and-remote-ssh-port-forwarding/
-N - Do not execute a remote command. This is useful for just forwarding ports. -f - background
5.21.1. Local
local port to conecting ssh mashine
-g Allows remote hosts to connect to local forwarded ports.
cons:
- TLS certificate validation fails
- When the website you are accessing redirects you to another URL, the connection fails because port forwarding is only valid for exactly this web server. This situation might be a problem when using single sign-on (SSO), for instance.
- ex
ssh -L 8080:localhost:80 root@host.com
localhost:8080 forwarded to remote address host.com:80.
Enables you to access another server from your local computer
- port numbers that are less than 1024 or greater than 49151 are reserved for the system
ssh -L 8080:www.slashdot.org:80 localhost
You will be prompted to enter your user password for the client machine. You are connecting, via SSH, back to your client machine, but creating the necessary SSH tunnel to the destination. After that, open up a browser and point it to http://localhost:8080. Your browser should automatically redirect you to Slashdot.
ssh -L 9090:example.com:80 admin@server.com
localhost port 9090 will be connected to example.com:80
ssh -L 9090:localhost:3306 admin@server.com
localhost 3306 will be opened as localhost 9090 for access admin@server.com
ssh -L 0.0.0.0:80:site.com:80 -p 9983 root@localhost
create mirror of site.com
5.21.2. Remote
require: GatewayPorts yes in sshd_config
ssh -R 9090:example.com:80 admin@server.com
server.com 9090 will be forwardet to example.com 80 which is accessed on local network only of server.com
ssh -R 5900:localhost:5900 USERNAME@192.168.1.192
192.168.1.192 5900 will be forwarded to localhost 5900 with encryption
ssh -R 6066:localhost:22 USERNAME@192.168.1.192
192.168.1.192 6066 will be forwarded to localhost 22 with encryption
5.21.3. Dynamic
ssh will act as SOCKS4/5 proxy (Port 1080 is the IANA registered port for SOCKS)
ssh -D 1080 bastion.securecorp.io
localhost 1080 is now SOCKS proxy to bastion.securecorp.io which is work as SOCKS proxy
5.21.4. examples
vps double socks
- ssh -D 1080 local_user@localhost # create socks proxy on localhost
- ssh -R 1080:localhost:1080 root@vps.com # allow access local 1080 port
- ssh -L 1080:localhost:1080 -p 49382 root@cps.com # crate local 1080 which will be forwarded to remote 1080
5.22. Reverse SSH Tunnel
Require: GatewayPorts yes in sshd_config ???
target:
- ssh -fN -R 7010:localhost:23 username@yourMachine
- yourMachine:7010 will be forwarded to target:22
https://jfrog.com/connect/post/reverse-ssh-tunneling-from-start-to-end/ https://www.marksanborn.net/howto/bypass-firewall-and-nat-with-reverse-ssh-tunnel/
5.23. USE CASES
compare private and public
- diff <( ssh-keygen -y -e -f "id_rsa" ) <( ssh-keygen -y -e -f "id_rsa.pub" )
login to directory
ssh root@host -t "cd /directory_wanted ; bash --login"
5.24. user with limited permissions
- /etc/ssh/sshd_config
Match Group sftp_users X11Forwarding no AllowTcpForwarding no ChrootDirectory /ubuntu # user folder ForceCommand internal-sftp
- /etc/passwd:
- ubuntu:x:1001:1001::/ubuntu:
- folder
- mkdir /ubuntu
- chown root.root ubuntu
5.25. links
5.26. ssh over socks5
- required: net-analyzer/openbsd-netcat
- ssh -o ProxyCommand='nc -X 5 -x 127.0.0.1:1081 %h %p' root@host
5.27. problems
Connection closed by UNKNOWN port 65535
- error in ProxyCommand
5.28. ssh as a tor hidden service
- apt update
- apt install tor
- server /etc/tor/torrc:
ShutdownWaitLength 1 seconds # ExcludeNodes {ru}, {cn}, {by}, {kz}, {sy}, {am}, {in} # paranoid GeoIPExcludeUnknown 1 # paranoid Sandbox 0 # not compatible with obfs4 Log notice file /var/log/tor/tor_notices.log SocksPort 0 ExitRelay 0 BridgeRelay 0 HiddenServiceDir /var/lib/tor/ssh HiddenServicePort 22 127.0.0.1:22
Steps:
- Client:
openssl genpkey -algorithm x25519 -out key.private.pem cat key.private.pem | grep -v " PRIVATE KEY" | base64 -d | tail --bytes=32 | base32 | sed 's/=//g' > private.key openssl pkey -in key.private.pem -pubout | grep -v " PUBLIC KEY" | base64 -d | tail --bytes=32 | base32 | sed 's/=//g' > public.key echo KEY_PUB=$(cat public.key)
- Server:
- <HiddenServiceDir>/authorized_clients/<user>.auth
- descriptor:x25519:<public.key>
- mkdir /var/lib/tor/ssh
- chmod go-a var/lib tor/ssh
- Client:
- torrc: ClientOnionAuthDir /var/lib/tor/userauth
- create file: <ClientOnionAuthDir>/<user>.auth_private
- <56-char-onion-addr-without-.onion-part>:descriptor:x25519:<private.key>
KEY_USER=ta ONION=xfe4swdndyhauaxj5di5ewnce2egr7qonn475vvopywonpe76jara2id.onion PRIVATE= cat <<EOF > userauth.sh #!/bin/bash echo "${ONION:0:56}:descriptor:x25519:$PRIVATE" > /var/lib/tor/userauth/$KEY_USER.auth_private EOF
SSH_TOR_PORT=88 SSH_PORT=49382 KEY_USER=ta KEY_PUB=P66JHL7W3R32O2DJBSGNVNR66JKJAS4KDG5Q4Q4EQM7CVH3X3IGQ apt update apt install tor cat <<EOF > /etc/tor/torrc RunAsDaemon 1 ShutdownWaitLength 1 seconds GeoIPExcludeUnknown 0 Sandbox 0 # not compatible with obfs4 Log notice file /var/log/tor/tor_notices.log SocksPort 0 ExitRelay 0 BridgeRelay 0 HiddenServiceDir /var/lib/tor/ssh HiddenServicePort $SSH_TOR_PORT 127.0.0.1:$SSH_PORT EOF mkdir -p /var/lib/tor/ssh/authorized_clients chmod -R 700 /var/lib/tor/ssh chown -R debian-tor:debian-tor /var/lib/tor/ssh echo "descriptor:x25519:$KEY_PUB" > /var/lib/tor/ssh/authorized_clients/$KEY_USER.auth systemctl restart tor sleep 7 cat /var/lib/tor/ssh/hostname
5.29. ssh systemd service
[Unit] Description=Setup a secure tunnel to %I After=network.target [Service] Environment="LOCAL_ADDR=localhost" EnvironmentFile=/etc/default/secure-tunnel@%i ExecStart=/usr/bin/ssh -NT -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -L ${LOCAL_ADDR}:${LOCAL_PORT}:localhost:${REMOTE_PORT} ${TARGET} # Restart every >2 seconds to avoid StartLimitInterval failure RestartSec=5 Restart=always [Install] WantedBy=multi-user.target
5.30. rsync over ssh
Copy from remote to local
- rsync required on remote host:
rsync -avzhP -e "ssh -p 6767" root@41.80.200.1:/home/rtorrent/downloads/Guns* /home/ssh/downloads/
6. jq - JSON parser
6.1. output format
By default, jq pretty-prints JSON output.
- –raw-output / -r raw JSON or string without quotes
- –raw-output0 new lines replace with NUL
- –compact-output / -c:
- –color-output / -C and –monochrome-output / -M:
- –indent n
- –tab - use tab instead of spaces
6.2. access
- .[] get element
- ".[] | .name" access field
- " | .name" map for every
- "map(.name)" map for every
- "| keys" function: get the keys of an object
- "| length" function: get length
- map(has("name")) check key exist and return true or false
- min/max
- unique
- jq '.[] | select(.color=="yellow")' - by condition
- .[] | select(.name|test("^o.")) - by regex match
omit one of the indexes:
echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[:6]' | jq '.[-2:]' # [ 5, 6 ]
echo '{"fruit1":{"name":"apple"},"fruit2":{"name":"orange"}}' | jq '.[] | select(.name|test("^o."))'
echo '{"fruit":{"name":"apple","color":"green","price":1.20}}' | jq -r 'map(has("name"))'
[ true ]
{ "result": { "alternatives": [ { "message": { "role": "assistant", "text": "Согласно документации" }, "status": "ALTERNATIVE_STATUS_FINAL" } ], "usage": { "inputTextTokens": "29", "completionTokens": "77", "totalTokens": "106" }, "modelVersion": "18.01.2024" } }
cat | jq "map(.alternatives)"
6.3. building
- —null-input / -n
BUCKET_NAME=testbucket OBJECT_NAME=testworkflow-2.0.1.jar TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar jq -n \ --arg bn "$BUCKET_NAME" \ --arg on "$OBJECT_NAME" \ --arg tl "$TARGET_LOCATION" \ '{bucketname: $bn, objectname: $on, targetlocation: $tl}'
7. print JSON
BUCKET_NAME=testbucket OBJECT_NAME=testworkflow-2.0.1.jar TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar JSON_FMT='{"bucketname":"%s","objectname":"%s","targetlocation":"%s"}\n' printf "$JSON_FMT" "$BUCKET_NAME" "$OBJECT_NAME" "$TARGET_LOCATION"