April 13, 2010
By Pierre Vignéras More stories by this author:

Abstract:

RAID has still not been adopted by most end-users despite its inherent quality such as performance and reliability. Reasons such as complexity of RAID technology (levels, hard/soft), set-up, or support may be given. We believe the main reason is that most end-users own a vast amount of heterogeneous storage devices (USB stick, IDE/SATA/SCSI internal/external hard drives, SD/XD Card, SSD, ...), and that RAID-based systems are mostly designed for homogenous (in size and technology) hard disks. Therefore, there is currently no storage solution that manages heterogeneous storage devices efficiently.

In this article, we propose such a solution and we call it PROUHD (Pool of RAID Over User Heterogeneous Devices). This solution supports heterogeneous (in size and technology) storage devices, maximizes the available storage space consumption, is tolerant to device failure up to a customizable degree, still makes automatic addition, removal and replacement of storage devices possible and remains performant in the face of average end-user workflow.

Although this article makes some references to Linux, the algorithms described are independent of the operating system and thus may be implemented on any of them.

Introduction

Whereas RAID¹ has been massively adopted by the industry, it is still not common on end-users desktop. Complexity of RAID system might be one reason... among many others. Actually, in a state-of-the-art data center, the storage is designed according to some requirements (the ''top-bottom'' approach already discussed in a previous article²). Therefore, from a RAID perspective, the storage is usually composed of a pool of disks of same size and characteristics including spares³. The focus is often on performance. The global storage capacity is usually not a big deal.

The average end-user case is rather different in that their global storage capacity is composed of various storage devices such as:

• Hard drives (internal IDE, internal/external SATA, external USB, external Firewire);
• USB Sticks;
• Flash Memory such as SDCard, XDCard, ...;
• SSD.

On the opposite, performance is not the big deal for the end-user: most usage does not require very high throughput. Cost and capacity are main important factors along with ease of use. By the way, the end-user does not usually have any spare devices.

We propose in this paper an algorithm for disk layout using (software) RAID that has the following characteristics:

• it supports heterogeneous storage devices (size and technology);
• it maximizes storage space;
• it is tolerant to device failure up to a certain degree that depends on the number of available devices and on the RAID level chosen;
• it still makes automatic addition, removal and replacement of storage devices possible under certain conditions;
• it remains performant in the face of average end-user workflow.

Algorithm

Description

Conceptually, we first stack storage devices one over the other as shown in figure 1.

Figure 1:Stacking storage devices (same size, ideal RAID case).

On that example with devices, each of capacity (terabytes), we end up with a global storage capacity of . From that global storage space, using RAID, you can get:

• a 4 Tb () virtual storage devices (called PV for Physical Volume⁴ in the following) using RAID0 (level 0), but then you have no fault tolerancy (if a physical device fail, the whole virtual device is lost).
• a 1 Tb () PV using RAID1; in that case, you have a fault tolerancy degree of 3 (the PV remains valid in the face of 3 drives failure, and this is the maximum).
• a 3 Tb () PV using RAID5; in that case, you have a fault tolerancy degree of 1;
• a 2 Tb () PV using RAID10; it that case, the fault tolerancy degree is also 1⁵ ( is the number of mirrored sets, 2 in our case).

There are more and more Laptops nowadays which do not have a CD/DVD-ROM facility build in but are able to boot from USB memory stick. This small guide provides all needs on how to create a bootable USB memory stick to install a Debian without a need for CD/DVD drive.

Before you continue make sure that you have backed up all your data from your USB memory stick. After completing this guide your USB memory stick will contain a new partition table and all your data will be lost.

This guide also assumes that you have a running linux system and USB memory stick with minimal size of 512MB which you will use for the process of creating Debian boot installation USB stick.

Find USB device name

Plug in your USB stick and find its device file name by:

fdisk -l

For the rest of this guide we are going to use /dev/sdb as our target.

WARNING: DO NOT COPY AND PASTE COMMANDS FROM THIS PAGE UNLESS YOUR TARGET DEVICE IS ALSO /dev/sdb !!

Create FAT16 partition

Next we need to create FAT16 partition. For this we can use cfdisk with -z option. This command will completely remove all your data from your USB stick, so backup first !:

cfdisk -z /dev/sdb

and create a first primary partition with the minimal size of 260MB and leave the rest unpartitioned or create second partition with some other filesystem for normal use of your USB stick. The size of 260 MB will fit a current Debian stable boot.img.gz and and Debian netinst ISO image.

Author: Lubos Rendek

Date: 21.03.2010

Name

touch [man page] - change file timestamps

Usage

When working with a Linux operating system, sometimes a need arises to create an empty file. Some service, which an administrator is configuring, requires a certain file to be created before the service can start and what is more this file may need to have certain access time stamp. If you are, for example, running a photo album web application which automatically adds your images to the album and sorts them according to the image creation time, the images you upload must have set accurate creation time in order this can work. What if the time set in your camera is incorrect?

A tool which comes handy in both scenarios is the touch command. Touch command not only provides a simple way to create files, but it also allows a user to alter access and modification time for a given file.

NOTE: Linux system does not store a file creation time. The information stored instead is the access time. The access time does not necessarily represent creation time and in many situations it represents access time only. This is because access time gets updated every time a file is accessed / read.

NOTE: In this article, we are going to use the stat command to observe changes to file's access and modification time made by touch command.

File creation

As it was already mentioned a touch command provides a simple way of creating empty files. The syntax is rather simple:

linuxconfig.org:~$ ls
linuxconfig.org:~$ touch fileA
linuxconfig.org:~$ ls
fileA
linuxconfig.org:~$

Execution of touch command with an argument fileA will produce a single empty file named fileA. Touch command can accept more than one argument and as result:

linuxconfig.org:~$ touch fileB fileC fileD
linuxconfig.org:~$ ls
fileA  fileB  fileC  fileD
linuxconfig.org:~$

it will create three additional files called fileB fileC and fileD. It is important to understand that all files created by touch command are empty, which means that touch command does not add any characters to newly created files.

Change file timestamps

At this point, we need to have a close look at earlier created files. stat command will return verbose information associated to a file given to it as its argument.

linuxconfig.org:~$ stat fileA
  File: `fileA'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d      Inode: 1867780     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/linuxconfig)   Gid: ( 1001/linuxconfig)
Access: 2010-03-21 06:22:52.000000000 +1100
Modify: 2010-03-21 06:22:52.000000000 +1100
Change: 2010-03-21 06:22:52.000000000 +1100
linuxconfig.org:~$

stat command returned detailed information about fileA status in regard to a present file system. The only interesting bit connected to this article is access and modify time.

Access time

Access time refers to a time when to file was last time accessed. In other words, every time some utility reads a content of fileA, fileA will change its access time. Following example demonstrates this point:

linuxconfig.org:~$ date
Sun Mar 21 06:23:08 EST 2010
linuxconfig.org:~$ stat -c %x fileA
2010-03-21 06:22:52.000000000 +1100
linuxconfig.org:~$ cat fileA
linuxconfig.org:~$ stat -c %x fileA
2010-03-21 06:23:19.000000000 +1100
linuxconfig.org:~$

First, we have used the date command to retrieve a current date and time. Next command executed is stat command which returns an access time of fileA. Cat command is used to access / read a content of fileA. There is no output since fileA is an empty file. Executing a stat command again with fileA as an argument indicates that access time has been changed.

Modify time

Modify time is altered when the content of a file itself has been modified. In the following example, we use echo command to modify a content of fileA:

linuxconfig.org:~$ date
Sun Mar 21 06:23:23 EST 2010
linuxconfig.org:~$ stat -c %y fileA
2010-03-21 06:22:52.000000000 +1100
linuxconfig.org:~$ echo addline > fileA
linuxconfig.org:~$ stat -c %y fileA
2010-03-21 06:23:35.000000000 +1100
linuxconfig.org:~$

Once again, we have used a date command to show a current system time. Next command executed was a stat command which in this case displayed modify time of a fileA. Next, we have used an echo command to add new line into fileA. Lastly, stat command confirms that modify time has been changed. As a side-note it should be mentioned that modifying content of file will alter also file's change time.

Change time

In regard to a touch command, we are not interested in change time. Nevertheless, just for a completeness we shortly examine also change time. Change time holds a time when file's meta data or inode information is altered. Meta data includes change of permissions or file ownership. Following example illustrates this idea:

linuxconfig.org:~$ date
Sun Mar 21 06:23:40 EST 2010
linuxconfig.org:~$ stat -c %z fileA
2010-03-21 06:23:35.000000000 +1100
linuxconfig.org:~$ chmod +x fileA
linuxconfig.org:~$ stat -c %z fileA
2010-03-21 06:23:51.000000000 +1100
linuxconfig.org:~$

Same as in both preceding examples date command displays a current date and time. stat command reveals a current value of file's change time. chmod command added executable permissions to a file. When comparing both time values, it is clear that a modification of fileA's permissions resulted in updating a change time value to a current system time.

Author: Lubos Rendek

Abstract

The purpose of this document is to help a reader to get started with Computer Vision library OpenCV on Linux system. OpencCV is a multi-platform library, but this article will be focused only on OpenCV using Linux operating system ( although, just the installation of the OpenCV library and video camera is platform-specific, all examples in this article should compile on any platform where OpenCV is properly installed such as  Mac OS, MS Windows and etc.).  Reader will be guided through a step-by-step guide on how to install and use some of the basic functions of OpenCV library such as displaying images, playing a video or using a video camera to process a video input stream.

Conventions used in this article:

• $ - execution on the command line by a non-privileged user
• # - execution on the command line by a superuser
• the actual command to be executed on the command line or code of program to be compiled
  
• OUTPUT: output produced on the command line by command execution
• NOTE: general notes and additional information

Introduction

In simple words a Computer Vision is a scientific field which attempts to provide a sight to the machine. This scientific field has expanded rapidly in recent years. Among researchers this growth is because of many improvements of vision algorithms and among the computer vision hobbyists this is due to the cheaper hardware components and processing power.  OpenCV library plays a great role in the Computer Vision field as it helps greatly to reduce cost and preparation time of computer vision research environment needed by university students, hobbyists and professionals. OpenCV also provides a simple to use functions to get the work done in a simple, effective and elegant manner.  OpenCV was started by Intel, and later it was transformed to an open source project now available on SourceForge.net. OpenCV library has multi-platform availability,  and it is partially written in C++ and C language. Despite the fact that this library is available on many Linux distributions from its relevant package repositories, in this article we will attempt to install and use OpenCV library compiled from a source code downloaded from SourceForge.net web site.


The reasons for compiling a source code may include:

• new version 2.0.0 recently released and more features available
• some bugs fixed which affected Linux OpenCV 1.0.0 versions ( such as cvGetCaptureProperty() etc. )
• more support is available for OpenCV 2.0.0 version than for former 1.0.0 version

This article will start with installation of OpenCV on Debian 5.0 ( Lenny ). Later a reader will be guided through a number of examples on how to use OpenCV to display an image, play a video and use camera to capture the video input stream.