Using jq to manipulate JSON in command line

Sometimes during the testing of REST API with curl in the command line, we need to process JSON from the response to present it in readable format or check a specific key or value. I used jq utility for that purpose and didn’t pay too much attention to how powerful it can be:

curl https://mysuperapi.com/superendpoint | jq

Recently, I found this article showed many cool examples of using jq. For example, to get a list of all sensors from this JSON:

[
    {
        "name": "Station1",
        "location": "Centerville",
        "sensors": ["temperature", "humidity"]
    },
    {
        "name": "Station5",
        "location": "Anytown",
        "sensors": ["temperature", "humidity", "rainfall", "wind_speed"]
    }
]

the following command can be used:

curl https://mysuperapi.com/superendpoint | jq -r '.[] .sensors[]'

The result will be like this:

temperature
humidity
temperature
humidity
rainfall
wind_speed

How to highlight SVN diff

By default Subversion shows diff without highlighting which makes a bit difficult review of big changes.

SVN

But that situation can be changed extremely simple and fast. I assume you use Fedora 17. For the other distributions you need just use different package manager, choose correct package name and find configs in the proper places.

First of all, install utility colordiff:
sudo yum install colordiff

Now you can get highlighted output of SVN diff:
svn diff foo.php | colordiff

SVN

But it isn’t convenient way. Since Subversion allows to use any external program to make a diff we just should just add following in the ~/.subversion/config:
diff-cmd = colordiff
And voilà, we can see colored diff just run ordinary SVN diff command:
svn diff foo.php

SVN

UPDATED:
To change colors in colordiff just copy its global config to your home directory and edit it:
cp /etc/colordiffrc ~/.colordiffrc
vim ~/.colordiffrc

How to get size of Mysql tables

In the previous post I wrote about a way to get a Mysql database size from Mysql shell. Now I’d like to share a way to get size of each Mysql table:
SELECT table_name,`engine` ,ROUND(data_length/1024/1024/1024,2) total_size_gb, ROUND(index_length/1024/1024/1024,2) total_index_size_gb, table_rows FROM information_schema.TABLES WHERE table_schema = 'rt3';
It’s useful if you need to know what you should clean in your huge database.

[via Techie-Gyan]

How to rebuild Ubuntu package from the sources

There are two ways (at least) to rebuild Ubuntu package from the source.
Using Ubuntu diff file and dpkg-buildpackage

  • Download source from Ubuntu repository.
  • Download Ubuntu diff file and apply it:
    patch -p1 < ubuntu.diff
  • Build package:
    dpkg-buildpackage -rfakeroot

Using .dsc file and debuild

  • Download .dsc file
  • Run following command:
    dpkg-source -x file.dsc
    Note, sometimes the archive with sources should be presented in the same directory!
  • Go to source directory and start building:
    debuild

In both cases the development packages should be installed first:
sudo apt-get install build-essential devscripts

Fixing GRUB bootloader on dual-boot PC

I have two versions of LinuxMCE on my HTPC – stable 0710 and developer 0810. The 0810 was installed after 0710. So, the GRUB bootloader was placed on 0810 partition. A few days ago I decided re-install 0810 from the scratch and I deleted its partition including GRUB booloader. As result I couldn’t boot 0710 anymore. Thanks to GRUB power and flexibility I was able to fix that problem and watch Champions League 🙂 The solution is simple and clear: boot from any LiveCD, drop terminal and type following command there:
sudo grub
>root (hd0,0)
>setup (hd0)

That’s it. After booting from HDD the GRUB menu was found again.

Log for Mysql console session

To log Mysql console session use option –tee (two dashes!) with full path to the log file:
mysql -uroot my_db --tee=/tmp/mysql_console.log
As result the file /tmp/mysql_console.log will contains all commands and queries with result of their executions. That might be helpful to keep your queries for using next time or for troubleshooting.

The author of the post, where I found that useful info, said that each time the Mysql session is started with logging the log file will be replaced. In my case it isn’t. As I expected the new messages just are appended to the log file.

VIM and Mysql integration

I use VIM with additional configuration instead of IDE. That configuration includes code explorer using ctag, syntax checking for PHP and Perl and run Perl scripts (syntax highlighting is not a subject of discussion ;)). Also I should run Mysql queries often. So, I decided to add integration with Mysql to VIM. I managed to do that thanks to that simple solution. Just add this code to your .vimrc:
to select database:
map <C-d> :call SwitchDB()<CR>
:function SwitchDB()
: let g:current_db = input("Database > ")
:endfunction

to run query:

map <C-m> :call Doquery()<CR>
:function Doquery()
: if !exists("g:current_db")
: call SwitchDB()
: endif
: let query_string = input(g:current_db . " > " )
: if query_string != ""
: exe "!mysql " . g:current_db . " -e \"" . escape(query_string, '"') . "\""
: endif
:endfunction

So, when you press Ctrl-d the VIM gives you a prompt to type database name, Ctrl-m will run query. Sure you can use the your preferable combinations of keys. Also you may extend the login functionality by adding prompt for database host and user or hardcoded that information in the SwitchDB() function.

Using VIM to construct queries in the Mysql shell even much easier. Just specify VIM as default editor:
export EDITOR=vim
After that type \e in the Mysql shell. It’ll bring you VIM window where you’ll able to type your query. To finish with editing just type ZZ as usual. To run built query type ; and press Enter.

As result we’ll have possibility to run Mysql queries directly from VIM and using VIM to create queries in the Mysql shell.

Add history and auto-complete to the SQL*Plus

I was very surprised how pure the Oracle’s command line utility SQL*Plus. Comparing with mysql it lack history, completion and doesn’t allow even edit query. To fix that the utility rlwrap can be used. I installed it via yum under Fedora 11 and run like that:
rlwrap sqlplus db_user@db
To have auto-complete feature you should find file SQL.dict somewhere (drop the comment, please, if you know the place). It contains SQL keywords, functions and commands. The command to run SQL*Plus will be following in that case:
rlwrap -b "" -f $HOME/sql.dict sqlplus db_user@db

To make our life easier we can create an alias in the .bash_profile or .profile:

alias mysqlplus='rlwrap -b "" -f $HOME/sql.dict sqlplus'

There is an another interesting improvement for Linux users – using VIM as default editor of SQL*Plus. I didn’t try it practically yet.

[via Oracle Online]