torsdag 23 juni 2011

archiving & Compressing

tar short for Tape Archive commonly referred to as tarball is both a file format and a program used to handle such files. The format was developed in the early days of Unix and made standard POSIX.1-1998 and later POSIX.1-2001.

Its easy to compress a whole directory in Unix/Linux´sBSD. Its useful when you are backing-up, transferring files over the Internet.

To make a tarball you need to follow the syntax and,

tar -zcvf archive_name directory_name


  • z compress a archive using gzip program
  • c create archive
  • v verbose - displays progress 
  • f archive file name
To de-compress a file follow this syntax,

tar -zxvf archive_name directory_name

where

  • x Extract files
 -----------------------------------
Read on


The basics of file compression in Unix
Why small files may be larger than large files. Why one file may be better than two, and how you can get more space on your disk by compressing your files using Tape Archiove (tar, tarball).

If you list files on your Unix system using ls -l you get the file size, however not the amount of space used on the disk. To se the about used by the disk you set a flag to -s, so ls -ls. This will display a initial number representing blocks used by the file. A block is a unit of 512 bytes, so 4 blocks would be 2048 bytes

The reason a filsystem uses blocks is that if it would be using the actual size would creata a overhead that  would become a load on the system. Also keep in mind that you need some information about the file, like name and other information. Also a file would become heavily fragmentet (parts of the file spread across the filesystem) leading to slow performance.

To handle the problem a compromise was reached in disk organization. A convenient number of bytes was selected as the minimum amount that could be allocated to a file, the allocation unit. If a file don´t use the allocated unit, this would be reserved for future expansion.

Early Unix system used 512 bytes, however this was expanded as memory increased to 1024 bytes on most Unix systems (larger on some), but many utilities (like ls) continues to report  in 512 bytes-blocks even though the actual size increased.

Black holes
With this in mind its easy to understand that a 3 byte file will occupy 512 bytes in the momory, 1-block on the disk. If the file is expanded over 512 bytes it will occupy another block though now using 1024 bytes on your disk.

This makes for a lot of space on your system. If you have a file that is 1201 bytes therefor occupy 3 blocks on a system that uses the 1024 byte-block 99.7 percent of the space allocated is unused, and 41.4 percent is wasted. Multiply this by the number of files on your system and you could imagean the vast black holes of disk space.

Remember that this kind of waste only occurs on very small files, so the larger files are more efficient in allocation space on your disk. If you allow that your system is probably working well your allocation is a good compromise between allocation and speed of disk access.

To establish the allocation unit of your system. You can read the man-page for the ls command. Establish the block size used by the -s flag.

Compression
Pack, compress, and gzip compression utilities work well on large files, but perform poorly on small files. In the sample below, compress is applied on the file and the result is displayed. The compress utility recognizes that it can´t do any good on the file and leaves it alone.  If it would to a do a job it would append the .z to a file when it compresses it.  To reverse the process use the uncompress command.

The smallest blocks is a pair of two (or the default allocation of units on your system).

Compressing files with tar
If you have a directory of small files that are little used, but need to remain on the system. One way of doing it is to combine the files and compress them. The obvious candidate for combining files are tar (tape archive) utility program.

Tar informs you that it has appended a file and how many tape blocks it uses. tar reprots in 512 bytes blocks.

Having a look at the files produced you might get a shock! The files may now take even more blocks than prior. Fortunately, tar fills those empty spaces in the blocks with garbage (takes form of hex zeros or NULLs.

Proceeding the next logical steps, is to compress those tarballs (tar). This results in a .tar.Z file that is smaller than the original file.

Just reverse the process starting with uncompress, and then use tar with the flag -xvf


It might be easier to just follow the tar command as described at the beginning of this guide. However if you would like to use archiving and compressing of your choice: That makes this process more powerful - and operating from the command line,  is that not why we prefer shells infront its desktop alternative? Now we can perform a tailored tarball for best performance of a particular file.   


söndag 19 juni 2011

FTP

The user interface to the Internet Standard File Transfer Protocol.

Its a program commonly used for copying files to and from systems on a network. These systems may be on the same site or anywhere in the world.

For the purpose of this tutorial, your machine is the local, where you are issuing the commands. The remote machine is the one holding the FTP server, the machine that performs the desired actions on its system and transfers the file to you.

Getting Started


Connecting to a remote machine, type


ftp machine_name.domain


Where machine_name is the full path to the server ex, domain.something.something
Or you could enter a IP-address, type


ftp 192.168.1.0


Sometimes you may which to access a system that you don´t have an account. You may then use the anonymous login_name (anonymous must be active on the system). Normally you are asked for your username and passwords. If you should type in the wrong data you can use the USER to relogin

USER


I would recommend that you compress directories when you want to transfer multiple files, it makes transfer easy and safe. Please follow the guide on tape archive (tar, tarball)


Finally here are the common commands with FTP


Common commands
? to request help or information about the
FTP commands
ascii to set the mode of file transfer to ASCII

(this is
the default and transmits seven bits per character)
binary to set the mode of file transfer to binary

(the
binary mode transmits all eight bits per byte and thus
provides less chance of a transmission error and must be
used to transmit files other than ASCII files)
bye to exit the FTP environment (same as
quit)
cd to change directory on the remote machine
close to terminate a connection with another computer
close brubeck closes the current FTP connection with brubeck,

  but still leaves you within the FTP environment.
delete to delete (remove) a file in the current remote
directory (same as rm in UNIX)
get to copy one file from the remote machine
to the local machine
get ABC DEF copies file ABC in the current remote directory
to (or on top of) a file named DEF in your
current local directory.
get ABC copies file ABC in the current remote directory
to (or on top of) a file with the same name,
ABC, in your current local directory.
help to request a list of all available
FTP commands
lcd to change directory on your local machine
(same as UNIX cd)
ls to list the names of the files in the current
remote directory
mkdir to make a new directory within the current
remote directory
mget to copy multiple files from the remote
machine to the local machine;

  you are prompted for a y/n
answer before transferring each file
mget * copies all the files in the current remote directory
to your current local directory, using the same
filenames. Notice the use of the wild card
character, *.
mput to copy multiple files from the local
machine to the remote machine;

  you are prompted for a y/n
answer before transferring each file
open to open a connection with another computer
open brubeck opens a new FTP connection with brubeck;

  you must enter a username and password
for a brubeck account

      (unless it is to be
an anonymous connection).
put to copy one file from the local machine to
the remote machine
pwd to find out the pathname of the current
directory on the remote machine
quit to exit the FTP environment (same as
bye)
rmdir to to remove (delete) a directory in the
current remote directory

fredag 17 juni 2011

Screeen

Screen is a GNU project program that will allow your CLI applications to run in the background without x running. Using the terminal to run many programs, it will fast get messy having all the terminal windows in view. This is easy to take care of by installing screen.

When screen is called, it creates a single window with a shell in it and then gets out of the way so that you can use the program as you normally would. At any time you can create a new window with other programs in them, kill the current, view scrollback, switch between windows, etc. As i stated before you programs still run when you are not viewing them. Everything you type is sent to the one keystroke that is used to initiate a command to the window manager. By default, each command begins with a control-a (abbriviated C-a from now on)

You will probably have screen installed on your system already if not, in Ubuntu do

$ sudo apt-get install screen

Then copy screenrc to your home directory

$ cp /etc/screenrc ~/.screenrc

If you would like to have screen always run with rtorrent (What I am using it for right now) add the following to your screenrc file

# Always start rtorrent with screen
screen -t rtorrent rtorrent


To start screen and rtorrent, simply run screen from a terminal . Control-a followed by d will detach screen and running screen -r will open screen again


Tip - Its common to use screen with ssh to use rtorrent on a remote machine. I will not get in to that just yet. Stay tuned for a article about ssh, x11 and the use of rtorrent on a remote machine.

rtorrent

rtorrent is a Bittorrent client, with emphasis on speed (thats what we love about the command line:). Even tho its faster than a GUI torrent client it dose not lack the features of one, its a full-blown client that suit your every need.

You install it in Ubuntu by issuing the command

$ sudo apt-get install rtorrent

To start your saved torrents issue the command

$ rtorrent Torrent/atorrent.torrent


Your torrent, both finished and unfinished will be saved in your home directory by default. To set another directory by using the -d flag

$ rtorrent -D Downloads/ Torrents/atorrent.torrent


I would however download the default config file and make the changes you want before you open rtorrent for the first time. You can get it here or use the wget command like the example below. Save it in your home directory as .rtorrent.rc. The reason I would never go without this is because I like to have a folder where I download torrents to and then have the client auto check for .torrent files in that directory


$ wget http://libtorrent.rakshasa.no/browser/trunk/rtorrent/doc/rtorrent.rc?format=raw


Open it with nano and configure rtorrents to suite your need, you get directions in the .rc file. Don´t forget to save it as rtorrent.rc!

$ nano /home/your_account/rtorrent.rc?format=raw

To make a directory that the client watches and adds, edit the rc file like bellow.


# Watch a directory for new torrents, and stop those that have been
# deleted.
#schedule = watch_directory,5,5,load_start=./watch/*.torrent
#schedule = untied_directory,5,5,stop_untied=
schedule = watch_directory,5,5,load_start=/home/[user]/watch/*.torrent
schedule = untied_directory,5,5,stop_untied=
schedule = tied_directory,5,5,start_tied=



Note - you need to mkdir the watch directory yourself.


Adding and removing torrents
Backspace -- Add torrent using an URL or file path. Use tab to view directory content and do auto-complete. Also, wildcards can be used. For example: ~/torrent/*
Return/Enter -- Same as backspace, except the torrent remains inactive. (Use ^s to activate)
CTRL-O -- Set new download directory for selected torrent. Only works if torrent has not yet been activated.
CTRL-S -- Start download. Runs hash first unless already done.
CTRL-D -- Stop an active download, or remove a stopped download.
CTRL-R -- Initiate hash check of torrent. Without starting to download/upload.

Throttling
a/s/d -- Increase the upload throttle by 1/5/50 KB.
z/x/c -- Decrease the upload throttle by 1/5/50 KB.
A/S/D -- Increase the download throttle by 1/5/50 KB.
Z/X/C -- Decrease the download throttle by 1/5/50 KB.

Navigating
Main View Keys:
right -- Switch to Download View.
CTRL-R -- Initiate hash check of torrent.
+/- -- Change priority of torrent.
l -- View log. Exit by pressing the space-bar.
1 -- Show all downloads.
2 -- Show all downloads, ordered by name.
3 -- Show started downloads.
4 -- Show stopped downloads.
5 -- Show complete downloads.
6 -- Show incomplete downloads.
7 -- Show hashing downloads.
8 -- Show seeding downloads.

Download View Keys
right -- Switch to selected view.
left -- Switch to view selection or back to main view.
1/2 -- Adjust max uploads.
3/4 -- Adjust min peers.
5/6 -- Adjust max peers.
p -- Display peer list.
o -- Display torrent info.
i -- Display file list.
u -- Display tracker list.
t/T -- Initiate tracker request. Use capital T to force the request, ignoring the "min interval" set by the tracker.

Peer list View Keys
left -- Switch to view selection.
right -- Show file details.
space -- Change the file priority; applies recursively when done on a directory.
* -- Change the priority of all files.
/ -- Collapse directories. While collapsed, press right to expand the selected directory.

Tracker list View Keys
left -- Switch to view selection.
* -- Enable/disable tracker.
space -- Rotate trackers in a group.



rtorrent wiki here