Storage‎ > ‎Storage-tape‎ > ‎

mhvtl - a linux virtual tape library

Thank you for visiting this page, this page has been update in another link mhvtl a linux virtual tape library
Well, I've been with tape library for many years, still remember I spent 3 days searching for a nonpropretary virtual tape library for my tape application development. I couldn't find one at that time, so I made pretty one just for my development. Now, I don't have to fight with test library anymore, but this one still caught my eyes, couldn't help not to try it. Great!

As it said on https://sites.google.com/site/linuxvtl2/

Emulations

Currently, only the tape module has 'personality modules'

For up-to-date view of emulations review vtltape.c and search for "static struct tape_drives_table"

list of tape emulations (as of v1.4-9)

Working emulations include:

    IBM LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)
    HP LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)
    IBM 03592-J1A, 03592-E05 & 03592-E06
    STK 9840 (A/B/C/D), 9940 (A/B)
    STK T10000 (A/B/C)
    Sony AIT (1/2/3 &4)

What about model X....

    Quantum (DLT/Super-DLT) is NOT supported as their SCSI Programmer's Guide does not document the expected returned data.
    DDS (4mm DAT) - I've not found any SCSI programmers' Guides for these drives.

Library emulation is coming in the future.

STK L180/L700 emulation works for NetBackup & NetWorker.

TSM - IBM library type needed to use LTO drives.

Note: Please let me know what works for which version of backup software so I can better document it.

Two packages are needed and they are in ELREPO on RHEL6/SL6 and CentOS. Though they are one version behind git, they stil look good.

kmod-mhvtl.x86_64                                                           1.3-1.el6.elrepo                                                          @elrepo
mhvtl-utils.x86_64                                                          1.3-1.el6.elrepo                                                          @elrepo

Just install them, lzo will be installed as well, then run
/etc/init.d/mhvtl start
By default, there are two libraries configured. I changed one to IBM TS3500, I put /etc/mhvtl/device.conf origional and new output here, you can see not much need to be changed.

<  Vendor identification: IBM
<  Product identification: 03584L32
origional one
>  Vendor identification: STK
>  Product identification: L700

Configuration change needs a restart to take effect, so after restart, it looks like this.

#lsscsi -g
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0   /dev/sg0
[2:0:0:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sda   /dev/sg1
[3:0:0:0]    mediumx IBM      03584L32         0103  /dev/sch0  /dev/sg10
[3:0:1:0]    tape    IBM      ULT3580-TD5      0103  /dev/st0   /dev/sg2
[3:0:2:0]    tape    IBM      ULT3580-TD5      0103  /dev/st1   /dev/sg3
[3:0:3:0]    tape    IBM      ULT3580-TD4      0103  /dev/st2   /dev/sg4
[3:0:4:0]    tape    IBM      ULT3580-TD4      0103  /dev/st3   /dev/sg5
[3:0:8:0]    mediumx STK      L80              0103  /dev/sch1  /dev/sg11
[3:0:9:0]    tape    STK      T10000B          0103  /dev/st4   /dev/sg6
[3:0:10:0]   tape    STK      T10000B          0103  /dev/st5   /dev/sg7
[3:0:11:0]   tape    STK      T10000B          0103  /dev/st6   /dev/sg8
[3:0:12:0]   tape    STK      T10000B          0103  /dev/st7   /dev/sg9

Check scsi VPD
#scsi_id --page 0x80 --whitelist --device=/dev/sg10
SIBM     03584L32       XYZZY_A  
#scsi_id --page 0x80 --whitelist --device=/dev/sg11
SSTK     L80            XYZZY_B  
#scsi_id --page 0x80 --whitelist --device=/dev/st0
SIBM     ULT3580-TD5    XYZZY_A1
#scsi_id --page 0x83 --whitelist --device=/dev/st0
350223344ab000100

Check device tree, it's from a pseudo adapter, you can't get it from lspci
#udevadm info -a --path=/class/scsi_tape/st0 | grep "looking at device"
  looking at device '/devices/pseudo_0/adapter0/host3/target3:0:1/3:0:1:0/scsi_tape/st0':
#udevadm info -a --path=/class/scsi_changer/sch0 | grep "looking at device"
  looking at device '/devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/scsi_changer/sch0':

I also tried to use stinit to initialized scsi2_logic for LTO tape drives, they are all succeed, support global blocking.
#mt -f /dev/fctd51 seek 0
#tar tvf /dev/fctd51
-rw-r--r-- vtl/vtl        3136 2013-08-30 06:24 ./device.conf
-rw-r--r-- vtl/vtl        3132 2013-08-30 06:23 ./device.conf.orig
-rw-r--r-- vtl/vtl        1373 2013-08-30 01:48 ./library_contents.10
-rw-r--r-- vtl/vtl        1453 2013-08-30 01:48 ./library_contents.30
-rw-r--r-- vtl/vtl         206 2013-08-30 01:48 ./mhvtl.conf
#mt -f /dev/fctd51 tell
At block 2.

Media changer works beautifully too

#mtx -f /dev/sg10 status
  Storage Changer /dev/sg10:4 Drives, 43 Slots ( 4 Import/Export )
Data Transfer Element 0:Full (Storage Element 30 Loaded):VolumeTag = F01030L5                           
Data Transfer Element 1:Empty
Data Transfer Element 2:Empty
Data Transfer Element 3:Empty
      Storage Element 1:Full :VolumeTag=E01001L4                           
      Storage Element 2:Full :VolumeTag=E01002L4                           
      Storage Element 3:Full :VolumeTag=E01003L4                           
      Storage Element 4:Full :VolumeTag=E01004L4                           
      Storage Element 5:Full :VolumeTag=E01005L4                           
      Storage Element 6:Full :VolumeTag=E01006L4                           
      Storage Element 7:Full :VolumeTag=E01007L4                           
      Storage Element 8:Full :VolumeTag=E01008L4       
...
#mtx -f /dev/sg10 unload 30 0
Unloading drive 0 into Storage Element 30...done

It doesn't emulate time cost which mechnical movement would have, so when you use it for development, count this factor in, other than that, nothing to complain.
 
In addition, in  https://sites.google.com/site/linuxvtl2, Mark said

License:
GPL v2
Q. Why GPL license.
A. Because I like the idea of sharing.
I don't profess to know everything (not even close). Having something I can share and allow others to share back is a good idea.
Virtual Tape Library consists of several components.

    LLD - A low level driver implemented as a kernel module - mhvtl
    Target devices - Daemons vtltape(1) and vtllibrary(1) which implement SCSI target device(s) in user-space
    Utility commands mktape(1), vtlcmd(1)
    And startup scripts build_library_config(1), make_vtl_devices(1)

The kernel module is based on the scsi_debug kernel module (http://sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note: As of 0.16, there are no default devices.

The support scripts will add Drives (SSC devices) & library (SMC) depending on the contents of /etc/mhvtl/library_contents and /etc/mhvtl/device.conf.

A char device back-end has been included with the vtl LLD driver This allows data and SCSI commands to be passed from the LLD to user-mode daemons (SCSI targets) which constently poll the driver and process any outstanding SCSI commands.
vtltape

vtltape(1) is the usermode SSC target daemon which writes/reads data to data files in the /opt/mhvtl directory (if a virtual tape has been loaded). The virtual tape files include a Medium Auxiliary Memory (MAM) data structure to store persistent data (number of tape loads, total data written/read, media type etc).
vtllibrary

vtllibrary(1) is the usermode SMC target daemon which reads its configuration from the file /etc/mhvtl/library_contents(5) at startup. The number of storage slots are built dynamically when the daemon starts. Hence changing the number of storage slots and media access slots are a matter of modifying the file contents and restarting the vtllibrary(1) daemon. All 'library' commands are performed on data structures in memory ONLY.
vtlcmd

A utility vtlcmd(1) is used to administrator the daemons vtltape(1) and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between vtlcmd(1), vtllibrary(1) and vtltape(1)

When a SCSI 'move medium' from a storage slot to a tape drive is requested, the media location is updated in vtllibrary(1) data structures, and the barcode of the media id is passed via the message queue to the vtltape(1) daemon in question. A file open of the barcode is attempted in the /opt/mhvtl directory and if successful, the vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit Ready requests. An unload SCSI command to the tape drive will close the data file.

Media can be moved out of the VTL via the Media Access Port. Once media is logically moved into the MAP slots, the MAP entries can be cleared using the vxcmd:
# vtlcmd library empty map

The media can be 'moved back again' by re-starting the VTL user-space daemons:

Media can be loaded into the MAP via a :

# vtlcmd library load map <barcode>

or restart the vtl

# /etc/init.d/mhvtl restart

TapeAlert

TapeAlert flags can be set using the vtlcmd(1)

e.g. vtlcmd [index] TapeAlert [flags]

Where index is the message Q offset associated with the drive (or the string 'library').

e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000
# vtlcmd 1 TapeAlert 80000

A listing of TapeAlert flags can be found at t10 home page for SSC devices Annex A.



Comments