Linux Ubuntu Bionic 18.04 - bash 4.4.20 | macOS 10.15.7 - bash 3.2.57 |
---|---|
eXtension of baSH
xsh is an extension of bash. It works as a bash library framework.
xsh aimed to provide a uniform and easy way to reuse bash code, like what a library does.
xsh - this repository, in a narrow sense, is not a library itself. It’s just a framework, in a broad sense xsh along with other repositories, such as xsh-lib/core
as a whole, is a bash library with a framework.
Why started this?
It is the only way I can write comfortable shellcode quickly for different purpose, and keep them well-organized as they grow, and be able to be reused at the time they are needed again.
This project is still at version 0.x
, and should be considered immature.
2. Design Philosophy
-
Reuse the shellcode previously written, provide a uniform way to organize the code, document the code, format the code, execute the code, even the way you write the code. As a result, some well-organized bash code libraries will be born.
-
Easy to bootstrap(install).
The only thing you need is a bash and Git client.
$ git clone https://github.com/alexzhangs/xsh $ bash xsh/install.sh $ . ~/.xshrc
-
Easy to invoke.
E.g. Call a utility named
upper
under the packagestring
in the libraryx
:$ xsh x/string/upper 'hello world' HELLO WORLD
-
Easy to understand how it works.
xsh framework, in technical, is just a single function, called
xsh()
, the only function insidexsh.sh
.Through
xsh()
, it’s able to load/unload libraries, call utilities of libraries, and get help.xsh libraries, are some Git repositories, hosted online, contain some
.sh
files, called utilities, present as functions and scripts, organized with some simple rules.You can build your libraries with your Git repositories.
-
Easy to make your existing code a library.
A xsh library is just a Git repo following some simple rules. The rules will be talked about later in the development section. Any repos following that rules can be able to load as xsh libraries.
2.1. What does xsh Framework Do?
- Provide an easy way to document and represent help and usage info
- Support basic logging
- Support debugging
- Support library versioning
- Handle the load/unload/update of libraries
2.2. What do xsh Libraries Do?
- Provide a container for utilities
- Provide rules for organizing utilities by packages
The xsh framework supports both public and private libraries.
2.3 What do xsh Utilities Do?
- Provide the fundamental function
- Provide help and usage information
- Handle temporary files
- Handle logs
- Handle output
Of course, some of the above topics can be built as libraries, packages, or utilities themselves.
3. xsh Bootstrap/Installation
The only thing you need is a bash and Git client.
$ git clone https://github.com/alexzhangs/xsh
$ bash xsh/install.sh
$ . ~/.xshrc
Or bootstrap xsh with a single line:
$ curl -s https://raw.githubusercontent.com/alexzhangs/xsh/master/boot | bash && . ~/.xshrc
4. xsh Usage
Before you can do something useful with xsh, you must load some libraries.
4.1. Load xsh Libraries
Use the xsh load
command to load a library.
Loading the latest tagged version of the library xsh-lib/core, you should issue:
$ xsh load xsh-lib/core
Cloning into '/Users/alex/.xsh/repo/xsh-lib/core'...
remote: Enumerating objects: 401, done.
remote: Counting objects: 100% (401/401), done.
remote: Compressing objects: 100% (237/237), done.
remote: Total 1276 (delta 149), reused 367 (delta 133), pack-reused 875
Receiving objects: 100% (1276/1276), 165.74 KiB | 18.00 KiB/s, done.
Resolving deltas: 100% (516/516), done.
Deleted tag '0.1.0' (was 5e7dcfb)
From https://github.com/xsh-lib/core
* [new tag] 0.1.0 -> 0.1.0
__xsh_git_force_update: INFO: Already at the latest version: 0.1.0.
Loading the latest development state of the library xsh-lib/core on default branch master
, you should issue:
$ xsh load -b master xsh-lib/core
After the lib is loaded, you can use the xsh list
command to list all loaded libraries.
$ xsh list
x (0.1.0) => xsh-lib/core
To list all the utilities of the library x
, use:
$ xsh list x
[script] x/log/filter
[functions] x/string/lower
[functions] x/string/random
[functions] x/string/upper
[functions] x/string/uuid
...
4.1.1. Update the Loaded xsh Libraries
Use the xsh update
command to update loaded libraries.
To update the previously loaded library xsh-lib/core
, simply issue:
$ xsh update xsh-lib/core
Deleted tag '0.1.0' (was 5e7dcfb)
From https://github.com/xsh-lib/core
* [new tag] 0.1.0 -> 0.1.0
__xsh_git_force_update: INFO: Already at the latest version: 0.1.0.
4.1.2. Unload the loaded xsh Libraries
Use the xsh unload
command to unload loaded libraries.
To unload the previously loaded library xsh-lib/core
, simply issue:
$ xsh unload xsh-lib/core
4.2. Invoke xsh Utilities
There are two methods to invoke xsh utilities.
Before to talk about that, let’s get familiar with the glossary LPUE
, LPUR
, and LPUC
.
-
LPUE stands for
Lib/Package/Util Expression
, a LPUE example for the libraryxsh-lib/core
isx/string/upper
. -
LPUR stands for
Lib/Package/Util Regex
, a LPUR example for the libraryxsh-lib/core
isx/string
that matches all the utilities under the package string. -
LPUC stands for
Lib/Package/Util Callable
, a LPUC example for the libraryxsh-lib/core
isx-string-upper
.
Now let’s get back to the three methods:
-
Call an individual LPUE.
The syntax:
$ xsh <LPUE> [options]
A sample:
$ xsh x/string/upper 'hello world' HELLO WORLD
-
Call a LPUC without the leading
xsh
command.The syntax:
<LPUC> [options]
In order to call a LPUC directly, you must import the LPUE first.
Use command
xsh imports
to import LPUEs. Afterward, you can call them directly as the syntax:<lib>-<package>-<util>
.A sample:
$ xsh imports x/string $ x-string-upper 'hello world' HELLO WORLD $ x-string-uuid 4be36c77-e507-4eee-9075-1aa259c1613e
4.3. Update xsh itself
See the current xsh version:
$ xsh version
0.1.4
List all available xsh versions(tags):
$ xsh versions
0.1.0
0.1.1
0.1.2
0.1.3
0.1.4
Update xsh to the latest tagged version:
$ xsh upgrade
Update xsh to a historical tagged version:
$ xsh upgrade -t <tag>
4.4. Get Help
See the help info of xsh itself:
$ xsh help
See the help info of xsh utilities by LPUR:
$ xsh help <LPUR>
See the specific section of help info of xsh utilities by LPUR:
$ xsh help -s <SECTION> <LPUR>
See the code of xsh utilities by LPUR:
$ xsh help -c <LPUR>
List all loaded libraries:
$ xsh list
x (0.1.0) => xsh-lib/core
List utilities by LPUR:
$ xsh list <LPUR>
List all utilities of all libraries:
$ xsh list '*'
5. Development
5.1. How to Make Your Own xsh Libraries?
The directory structure and files of a sample library look like this:
xsh-lib-sample/
├── functions
│ └── string
│ └── lower.sh
│ └── random.sh
│ └── upper.sh
│ └── uuid.sh
├── scripts
│ └── log
│ └── filter.sh
└── xsh.lib
Let each of your functions be a single .sh
file, named as the same as the function name, put them under the directory functions
, you are free to organize the subdirectories, for this sample, they are string
and log
that is called packages
.
5.1.1. xsh.lib
xsh.lib
is a config file for the library, xsh
will read configuration from it.
Supported configurations:
name=
-
Required: YES
-
Description:
is used as library name.
5.1.2 Sample code
cat xsh-lib-sample/xsh.lib
:
name=smpl
cat xsh-lib-sample/functions/string/upper.sh
:
#? Usage:
#? @upper STRING ...
#?
#? Output:
#? Uppercase presentation of STRING.
#?
#? Example:
#? @upper Foo
#? # FOO
#?
function upper () {
echo "$@" | tr [a-z] [A-Z]
}
You will need to follow the exact comment style to let xsh generate help info.
The function should be started with the exact syntax:
function <name> ()
It’s pretty the same with script files, except that you don’t have to define functions inside the script.
5.1.3. Commit the code and test it
Push the code to a Git repo, for example, GitHub, on branch master
, then the library is ready for the test.
Load the sample library xsh-lib-sample
on GitHub:
Note: Option -b master
is necessary to tell that you are loading the latest untagged version for testing purposes.
$ xsh load -b master <yourusername>/xsh-lib-sample
Or if the Git repo isn’t on GitHub, issue:
$ xsh load -s https://yourgitserver.com -b master <yourusername>/xsh-lib-sample
Then can be called as:
$ xsh smpl/string/lower
$ xsh smpl/string/upper
$ xsh smpl/log/filter
5.1.4. Publish the library
To tell the world that the library is ready, you have to make at least one Git tag to publish the library.
To use Semantic Versioning for the tag name is recommended.
To make an annotated tag 1.0.0
on the latest commit of branch master
, issue:
$ git tag -a -m 'v1.0.0' 1.0.0
Then the library is ready.
Load the published sample library xsh-lib-sample
on GitHub:
$ xsh load <yourusername>/xsh-lib-sample
5.1.5. Create library with template
Using the GitHub template repository xsh-lib/template is the simplest way to create a new empty xsh library.
5.1.6. xsh library INIT files
xsh library INIT files are used to initialize the library environment.
The INIT file is a script file named __init__.sh
and placed at any level under the library functions
directory.
The INIT file is sourced while importing any function utility, right before the function utility was sourced.
The source of the INIT file won’t happen again on the subsequence calls of the function utility until it is imported again, except a runtime
decorator is used on the INIT file.
5.1.7. xsh library decorators
Decorators are used to add metadata to the functions, and init files in xsh libraries.
It should start with @
and be placed at the comment block of the function, or INIT file, right before the function, or INIT file definition.
5.1.7.1. Decorators for functions
The decorators are used to add metadata to the functions in xsh libraries.
5.1.7.1.1. Decorator xsh
The xsh
decorator is used to call the xsh framework functions.
5.1.7.1.2. Decorator subshell
The subshell
decorator is used to create a subshell to isolate the environment of the function from the caller’s environment.
5.1.7.2. Decorators for INIT files
The decorators are used to add metadata to the INIT files in xsh libraries.
5.1.7.2.1. Decorator static
The static
decorator is used to make the INIT file to be sourced only once on the first call of the function utility, and won’t be sourced again on the subsequence calls of the function utility.
This is default behavior if no decorator is used on the INIT file.
5.1.7.2.2. Decorator runtime
The runtime
decorator is used to make the INIT file to be sourced on every call of the function utility, even it is already sourced before.
5.1.7.3. Examples of using decorators in xsh libraries
- Clean up on function return
#? @xsh imports /trap/return
#? @subshell
#?
function foo () {
x-trap-return -f "${FUNCNAME[0]}" "echo 'clean up on this function returns'"
echo "foo"
}
The equvalent code:
function foo () {
(
function __foo__ () {
xsh imports /trap/return
x-trap-return -f "${FUNCNAME[0]}" "echo 'clean up on this function returns'"
echo "foo"
}
__foo__ "$@"
)
}
see details by xsh help /trap/return
.
- Clean up on function return or any error occurs
#? @xsh imports /trap/return
#? @xsh /trap/err -rE
#? @subshell
#?
function foo () {
x-trap-return -f "${FUNCNAME[0]}" "echo 'clean up on this function returns or any error occurs'"
echo "foo"
}
see details by xsh help /trap/err
.
5.2. Debugging (Debug Mode)
With the debug mode enabled, the shell options: -vx
is set for the debugging utilities. The debug mode is available only for the commands started with xsh
.
Enable the debug mode by setting an environment variable: XSH_DEBUG
before the command xsh
.
Values for XSH_DEBUG:
1 : Enable the debug mode for whatever the LPUE input by `xsh`.
e.g: XSH_DEBUG=1 xsh /string/upper foo
<LPUR>: Enabled the debug mode for the LPUE input by `xsh` if the LPUE equals to or matches the <LPUR> set by XSH_DEBUG.
e.g: XSH_DEBUG=/string xsh /string/upper foo
e.g: XSH_DEBUG=/string/pipe/upper xsh /string/upper foo
The debug mode applies to the following commands and internal functions:
- calls
- call, exec
The debug mode is for debugging xsh libraries.
For the general debugging purpose, use xsh debug
, see xsh help debug
.
xsh debug [-1 OPTION] [-0 OPTION] [...] <FUNCTION | SCRIPT>
It provides a consistent way to debug functions and scripts without having to manually switch the shell options on and off, especially with functions, the syntax bash -x script.sh
is out of hands. Although you may use the subprocess syntax (set -x; foo_func)
to avoid messing the current process up, but subprocess has its own side effects.
5.3. Development at Local (Dev Mode)
The dev mode is for developers to develop xsh libraries.
With the dev mode enabled, the utilities from the development library will be used rather than those from the normal library.
The dev mode is available only for the commands started with xsh
.
Before using the dev mode, you need to create symbol links for the libraries that need to use dev mode, put the symbol links in the directory ~/.xsh/lib-dev
, and point them to your development workspaces.
This can be done with the command: xsh lib-dev-manager link ...
, and be undone with the command xsh lib-dev-manager unlink ...
.
Example:
$ xsh lib-dev-manager link xsh-lib/core ~/projects
$ xsh lib-dev-manager link xsh-lib/aws ~/projects
After the link, the development libraries look like:
$ ls -l ~/.xsh/lib-dev
total 0
lrwxr-xr-x 1 alex staff 32 Sep 4 16:36 aws -> /Users/alex/projects/xsh-lib/aws
lrwxr-xr-x 1 alex staff 33 Sep 4 16:36 x -> /Users/alex/projects/xsh-lib/core
Then the dev mode is ready to use.
Enable the dev mode by setting an environment variable: XSH_DEV
before the command xsh
.
Values for XSH_DEV:
1 : Enable the dev mode for whatever the LPUE or LPUR input by `xsh`.
e.g: XSH_DEV=1 xsh /string/upper foo
XSH_DEV=1 xsh import /string
XSH_DEV=1 xsh list
<LPUR>: Enabled the dev mode for the LPUE or LPUR input by `xsh` if the LPUE/LPUR equals to or matches the <LPUR> set by XSH_DEV.
e.g: XSH_DEV=/string xsh import /string
e.g: XSH_DEV=/string xsh help /string/upper
e.g: XSH_DEV=/string/pipe/upper xsh /string/upper foo
Be noted, the following usage won't work as expected:
e.g: XSH_DEV=/string xsh import /
The dev mode applies to the following commands and internal functions:
- calls, imports, unimports, list, help
- call, import, unimport, lib_list, help_lib
5.4. Development of xsh
- IDE: PyCharm & Aquamacs
- Code static analysis: ShellCheck
- Code static analysis integration: CodeFactor
- Code testing framework: ShellSpec
- Code coverage: Kcov
- Code coverage analysis: CodeCov
- Code hosting: GitHub
- CI hosting: Travis
Most of them are free, or free to OSS projects. Many thanks to whoever contributes to them.
6. Where to find xsh Libraries
-
Check out the repositories under official xsh library site.
-
Search GitHub repositories with the keyword
xsh-lib-
.
7. TODO
-
Dependency
-
Library registration
-
Document generation