Using jq to manipulate JSON in command line

Sometimes during the testing of REST API with curl in the command line, we need to process JSON from the response to present it in readable format or check a specific key or value. I used jq utility for that purpose and didn’t pay too much attention to how powerful it can be:

curl https://mysuperapi.com/superendpoint | jq

Recently, I found this article showed many cool examples of using jq. For example, to get a list of all sensors from this JSON:

[
    {
        "name": "Station1",
        "location": "Centerville",
        "sensors": ["temperature", "humidity"]
    },
    {
        "name": "Station5",
        "location": "Anytown",
        "sensors": ["temperature", "humidity", "rainfall", "wind_speed"]
    }
]

the following command can be used:

curl https://mysuperapi.com/superendpoint | jq -r '.[] .sensors[]'

The result will be like this:

temperature
humidity
temperature
humidity
rainfall
wind_speed

How to get size of Mysql tables

In the previous post I wrote about a way to get a Mysql database size from Mysql shell. Now I’d like to share a way to get size of each Mysql table:
SELECT table_name,`engine` ,ROUND(data_length/1024/1024/1024,2) total_size_gb, ROUND(index_length/1024/1024/1024,2) total_index_size_gb, table_rows FROM information_schema.TABLES WHERE table_schema = 'rt3';
It’s useful if you need to know what you should clean in your huge database.

[via Techie-Gyan]

How to check Mysql database size

Recently I needed to know the size of our RT3 database. So, I found this SQL to do that from Mysql shell:

SELECT table_schema "rt3", sum( data_length + index_length ) / 1024 / 1024 / 1024 "Data Base Size in GB" FROM information_schema.TABLES GROUP BY table_schema;

The result is 360GB! We have to cleanup before upgrade to RT4. Otherwise the upgrade procedure will take ages.

[via Mysql Forum]

Add history and auto-complete to the SQL*Plus

I was very surprised how pure the Oracle’s command line utility SQL*Plus. Comparing with mysql it lack history, completion and doesn’t allow even edit query. To fix that the utility rlwrap can be used. I installed it via yum under Fedora 11 and run like that:
rlwrap sqlplus db_user@db
To have auto-complete feature you should find file SQL.dict somewhere (drop the comment, please, if you know the place). It contains SQL keywords, functions and commands. The command to run SQL*Plus will be following in that case:
rlwrap -b "" -f $HOME/sql.dict sqlplus db_user@db

To make our life easier we can create an alias in the .bash_profile or .profile:

alias mysqlplus='rlwrap -b "" -f $HOME/sql.dict sqlplus'

There is an another interesting improvement for Linux users – using VIM as default editor of SQL*Plus. I didn’t try it practically yet.

[via Oracle Online]

Perl Oneliner: Recursive Search and Replace

I like Perl and I use it every day. It’s cool to make a simple scripts in one line to do some routine work. For example, my first Perl oneliner formats Apache log to be easy readable. Here is another good example of Perl onliner (it isn’t my actually). It makes a recursive search and replacement:
Perl -p -i -e 's/oldstring/newstring/g' `grep -ril oldstring *`

[via About: Perl]

Run PHP script from command-line

I came across a brief howto “Implementing with PHP: Standalone Scripts”. I’m Perl guy but I have to use PHP in command line to perform some SugarCRM backgound tasks. I found useful the part about parsing command-line arguments with Console/Getopt.php. It may do administration tasks much easier.

tar: Howto Exclude or Include Files

Recently I’ve had a little problem. I’ve needed to create an archive from some source directories but I’ve not needed to add in the archive some subdirectorives. I’ve made a quick search in Google and found an article “Telling tar Which Files to Exclude or Include”. Here is a short example:


$ find videoguide/ ! -type d -print | egrep '/,|%$|~$|.jpg$|.gif$|.png$' > /tmp/exclude_files

That command forms a list of excluded files and store it into temporary file.

$ tar vcfX ~/projects/arc/vg-19012004.tar /tmp/exclude_files

will remove files which listed into excluded_files from archive.

Updated: There is a more simple way to exclude file/dir from the archive. You can define a pattern in the command line instead of creation a file. Let’s image that we need to exclude Subversion directories from the archive:
tar vzcf my-project.tar.gz --exclude='.svn' ProjectDir
Note: you should always wrap the pattern by quotes!