Monday, August 23, 2010
a contribution
It seems a little patch I sent to the btpd project more than a year ago is finally accepted: commit, original message
Thursday, April 22, 2010
Evolution of uguu scanning process
Scanning is very essential part of a search engine. It is about how database is filled and updated. You can't search if nothing was scanned before. In this post I'd like to cover how scanning in uguu is implemented and how it has evolved over time.
From the very start it was decided that scanning should be split between two programs. The first one connects to a share, somehow constructs a list of contents of the share and prints it to the standard output. The second one is protocol-independent script which reads contents from the standard input and executes a sequence of SQL commands to insert contents into the database. This sounded very reasonable that time since I had already written smbscan in C and I didn't want to deal with SQL in C. Python appeared to be easier for this task. Also, simple text output helps a lot with testing and debugging.
smbscan itself also consisted of protocol dependent part (connecting to a share, go up, go to subdir, readdir) and independent one (constructing file tree). When Radist (the other one uguu developer) wanted to port his ftp scanner to uguu I asked him to use tree constructor from smbscan so we could change scanner output format more easily if necessary. Later this common part evolved into libuguu.
Now it's time to turn back to the scanning process. At first we thought that when time to scan a share comes all contents of the share should be dropped from the database and the share itself should be rescanned by protocol-dependent scanner having it's output read by the database updating script. If some error occurs SQL transaction rollback is executed and scanning process is marked as failed. If the scanner report success then changes are committed to the database. All described process is done inside a single SQL transaction so there is no need to care about what our users see in the middle of scanning process - they see old version until new one is committed.
When we first launched uguu it appeared scanning was very slow. Some optimizations were required. First idea came to my mind was not to perform database updating if contents of a share were unchanged. I thought to execute diff on scanner outputs saved to files, but Radist suggested comparing only security hash digests. We started to keep SHA1 digest of the scanner output in the database and checked against it before each update. Since many shares don't change their contents often uguu gained a big speedup here.
By that time I thought "If it were possible to diff old and new scanner outputs and commit only actual changes...". But scanning speed wasn't the biggest problem anymore - we needed to improve search time next. At the beginning of April database schema change was committed and scanning became major issue again.
Next long-awaited feature was automatic recursion detection in the tree constructor. The source of the problem is that filesystem links are not always detectable by the client. Suppose you have a directory A and you create a symbolic link B in it pointed to A. Then SMB protocol client would see that B is just a plain directory and it has a subdirectory B and so on infinitely. If A has a huge subtree then it is copied each time SMB client enters B resulting very huge scanner output and of course big database workload. Solution was to calculate MD5 digest for each directory and compare against it's ancestors. If a match occurs then directory parent's MD5 is compared to that ancestor's MD5. Recursion is said to be detected if along the path to the root we find a chain of directories which MD5s are the same as of such chain moved to the end of the path (current directory) and overall number of items in that chain's directories is bigger than a certain threshold. The latter condition provides a scale for a chain size: from 1 if linked directory is large to the threshold if there is a directory with only one child - the link itself.
Recursion detection is very useful feature if there are shares with recursive links. But there are very few such shares in our environment.
The next idea was quite simple to implement: if two shares have the same SHA1 digest then keep only one copy of contents in the database. This was supposed to reduce amount of files in the database for hosts with same contents via both SMB and FTP. But in reality this wasn't very helpful since some changes can occur between SMB and FTP scanning and which is more frustrating when a filename is not good enough it is shown differently via SMB and FTP. The latter argument kicks the whole idea to the ground.
So the only last hope remained - to construct a diff against old contents. Compared to previous work this was a big task with a huge amount of new code in libuguu but it is finally done and currently is being tested on our installation.
Tree diffing is very straightforward: first the whole old tree is reconstructed from a file, then while constructing a new tree using protocol-dependent walker, tree constructor compares new contents against old ones. If a new directory is added (respectively an old one is deleted) tree constructor prints the whole subtree with prefix '+' (respectively '-'). Same goes for files. If only size or number of items in a directory (or just size in case of a file) has been changed then only the directory (file) is printed with '*' modifier. Strictly speaking, output is a bit more complex to help further processing by the database updating script.
Dullness of the algorithm can be viewed in a situation when just some directory's name has been changed and that directory has a huge subtree. Then two trees will be presented in diff (one with '-' prefix and the other with '+'). However in uguu we keep tsvector of entire path for each file to allow searching in full paths. So we would have to update the entire subtree anyway.
Now uguu has scanners for FTP, SMB and WebDAV (the latter was written last weekend and is still quite unreliable). All of them use the tree constructor from libuguu and thus have all the features described above.
PS. Thanks Radist for pointing out some errors.
From the very start it was decided that scanning should be split between two programs. The first one connects to a share, somehow constructs a list of contents of the share and prints it to the standard output. The second one is protocol-independent script which reads contents from the standard input and executes a sequence of SQL commands to insert contents into the database. This sounded very reasonable that time since I had already written smbscan in C and I didn't want to deal with SQL in C. Python appeared to be easier for this task. Also, simple text output helps a lot with testing and debugging.
smbscan itself also consisted of protocol dependent part (connecting to a share, go up, go to subdir, readdir) and independent one (constructing file tree). When Radist (the other one uguu developer) wanted to port his ftp scanner to uguu I asked him to use tree constructor from smbscan so we could change scanner output format more easily if necessary. Later this common part evolved into libuguu.
Now it's time to turn back to the scanning process. At first we thought that when time to scan a share comes all contents of the share should be dropped from the database and the share itself should be rescanned by protocol-dependent scanner having it's output read by the database updating script. If some error occurs SQL transaction rollback is executed and scanning process is marked as failed. If the scanner report success then changes are committed to the database. All described process is done inside a single SQL transaction so there is no need to care about what our users see in the middle of scanning process - they see old version until new one is committed.
When we first launched uguu it appeared scanning was very slow. Some optimizations were required. First idea came to my mind was not to perform database updating if contents of a share were unchanged. I thought to execute diff on scanner outputs saved to files, but Radist suggested comparing only security hash digests. We started to keep SHA1 digest of the scanner output in the database and checked against it before each update. Since many shares don't change their contents often uguu gained a big speedup here.
By that time I thought "If it were possible to diff old and new scanner outputs and commit only actual changes...". But scanning speed wasn't the biggest problem anymore - we needed to improve search time next. At the beginning of April database schema change was committed and scanning became major issue again.
Next long-awaited feature was automatic recursion detection in the tree constructor. The source of the problem is that filesystem links are not always detectable by the client. Suppose you have a directory A and you create a symbolic link B in it pointed to A. Then SMB protocol client would see that B is just a plain directory and it has a subdirectory B and so on infinitely. If A has a huge subtree then it is copied each time SMB client enters B resulting very huge scanner output and of course big database workload. Solution was to calculate MD5 digest for each directory and compare against it's ancestors. If a match occurs then directory parent's MD5 is compared to that ancestor's MD5. Recursion is said to be detected if along the path to the root we find a chain of directories which MD5s are the same as of such chain moved to the end of the path (current directory) and overall number of items in that chain's directories is bigger than a certain threshold. The latter condition provides a scale for a chain size: from 1 if linked directory is large to the threshold if there is a directory with only one child - the link itself.
Recursion detection is very useful feature if there are shares with recursive links. But there are very few such shares in our environment.
The next idea was quite simple to implement: if two shares have the same SHA1 digest then keep only one copy of contents in the database. This was supposed to reduce amount of files in the database for hosts with same contents via both SMB and FTP. But in reality this wasn't very helpful since some changes can occur between SMB and FTP scanning and which is more frustrating when a filename is not good enough it is shown differently via SMB and FTP. The latter argument kicks the whole idea to the ground.
So the only last hope remained - to construct a diff against old contents. Compared to previous work this was a big task with a huge amount of new code in libuguu but it is finally done and currently is being tested on our installation.
Tree diffing is very straightforward: first the whole old tree is reconstructed from a file, then while constructing a new tree using protocol-dependent walker, tree constructor compares new contents against old ones. If a new directory is added (respectively an old one is deleted) tree constructor prints the whole subtree with prefix '+' (respectively '-'). Same goes for files. If only size or number of items in a directory (or just size in case of a file) has been changed then only the directory (file) is printed with '*' modifier. Strictly speaking, output is a bit more complex to help further processing by the database updating script.
Dullness of the algorithm can be viewed in a situation when just some directory's name has been changed and that directory has a huge subtree. Then two trees will be presented in diff (one with '-' prefix and the other with '+'). However in uguu we keep tsvector of entire path for each file to allow searching in full paths. So we would have to update the entire subtree anyway.
Now uguu has scanners for FTP, SMB and WebDAV (the latter was written last weekend and is still quite unreliable). All of them use the tree constructor from libuguu and thus have all the features described above.
PS. Thanks Radist for pointing out some errors.
Monday, February 8, 2010
simple port scanner
Strange, but well-known scanning tool nmap has a quite aggressive license. It is in fact GNU GPL Version 2 but with some exceptions. Among others there is one which may force you not to use nmap in your scripts:
This is very confusing and I've decided to write my own port scanner for uguu which is BSD-licensed. Needless to say, we planned to use nmap originally. With having our own tool we strike several goals at once: no nmap license terms violation, no external program execution in our python scripts, lessen the number external parts uguu is depended on, scan a list of host-port pairs at once without grouping by ports (you can give nmap list of hosts through stdin but you can't specify ports there, afaik). And with all these advantages it was fine by me to have a scanner with less performance. But as for my small /24 network with 4 computers and no crappy firewalls it is even faster than nmap (nmap -PS -p): 3 seconds against 8 seconds.
The scanner is written in python and is available as a part of uguu codebase (bin/network.py). It can be executed as a standalone application with ip range (three formats are available: single host, ip1-ip2, ip/netmask) and port specified in command line. Ip range can't be large for now due to number of open files limitations, but it worked for me with /24 netmask.
PS. Should I pass some other options to nmap? But no cheating here, it has to be a user-invoked port scanner.
* Note that the GPL places important restrictions on "derived works", yet *
* it does not provide a detailed definition of that term. To avoid *
* misunderstandings, we consider an application to constitute a *
* "derivative work" for the purpose of this license if it does any of the *
* following: *
...
* o Executes Nmap and parses the results (as opposed to typical shell or *
* execution-menu apps, which simply display raw Nmap output and so are *
* not derivative works.) *
This is very confusing and I've decided to write my own port scanner for uguu which is BSD-licensed. Needless to say, we planned to use nmap originally. With having our own tool we strike several goals at once: no nmap license terms violation, no external program execution in our python scripts, lessen the number external parts uguu is depended on, scan a list of host-port pairs at once without grouping by ports (you can give nmap list of hosts through stdin but you can't specify ports there, afaik). And with all these advantages it was fine by me to have a scanner with less performance. But as for my small /24 network with 4 computers and no crappy firewalls it is even faster than nmap (nmap -PS -p): 3 seconds against 8 seconds.
The scanner is written in python and is available as a part of uguu codebase (bin/network.py). It can be executed as a standalone application with ip range (three formats are available: single host, ip1-ip2, ip/netmask) and port specified in command line. Ip range can't be large for now due to number of open files limitations, but it worked for me with /24 netmask.
PS. Should I pass some other options to nmap? But no cheating here, it has to be a user-invoked port scanner.
Monday, January 11, 2010
uguu - search engine for local networks
I've spent new year holidays on designing and writing (alongside with another one developer) a local searcher which would allow quick search and browsing through SMB and FTP shares. For years we were using http://sourceforge.net/projects/fsmbsearch but I don't like neither it's design which almost doesn't rely on relational database (I may be wrong here and it would be interesting to find out the answer) nor it's implementation when installation requires dealing with autohell, patching some outdated version of samba and getting loads of perl modules.
The new search engine called uguu is entirely open source and is available at http://code.google.com/p/uguu. It uses libsmbclient and small amount of C code for scanning SMB share, postgres for database, python for scripts and django as web framework.
It is still very far way to go (for example, no search page yet) and there are some things I plan to do myself, but if you want to contribute in any way feel free to join google group devoted to this project: http://groups.google.com/group/uguu
The new search engine called uguu is entirely open source and is available at http://code.google.com/p/uguu. It uses libsmbclient and small amount of C code for scanning SMB share, postgres for database, python for scripts and django as web framework.
It is still very far way to go (for example, no search page yet) and there are some things I plan to do myself, but if you want to contribute in any way feel free to join google group devoted to this project: http://groups.google.com/group/uguu
Saturday, October 31, 2009
Spy mode inotify
Although linux inotify(7) user-space interface allows to catch events on file operations it doesn't provide a way to answer a simple question: who the hell is accessing my files. By 'who' I mean which process, of course. So if we want to get an answer for that question we have to go into kernel space. (Constantly doing 'lsof|grep' is out of question here.)
Little thinking give us a hope this can be done with quite simple kernel module. Indeed, inotify handler must be called just in the moment a file is accessed, meaning inotify handler is running in context of the process which is accessing a file. So our inotify handler can just output to dmesg desired fields from 'current' task. This should work until some sophisticated buffering is introduced between file operations and inotify handlers.
Kernel provides an easy way to use self-written inotify handler in a kernel module. See Documentation/filesystems/inotify.txt for details.
The code below just exhibits the described approach works. For trying it do
Then access that file and watch dmesg for messages like the following
spy_inotify.c
Little thinking give us a hope this can be done with quite simple kernel module. Indeed, inotify handler must be called just in the moment a file is accessed, meaning inotify handler is running in context of the process which is accessing a file. So our inotify handler can just output to dmesg desired fields from 'current' task. This should work until some sophisticated buffering is introduced between file operations and inotify handlers.
Kernel provides an easy way to use self-written inotify handler in a kernel module. See Documentation/filesystems/inotify.txt for details.
The code below just exhibits the described approach works. For trying it do
insmod spy_inotify.ko path=file_to_spy_on
Then access that file and watch dmesg for messages like the following
[spy_inotify] Catched inotify event for file ../../cpufreq.c. mask=32, pid=8183 executable="less"
spy_inotify.c
/* spy_inotify - linux kernel `module for spying on inotify events
* prints to dmesg _who_ exactly is doing something with a file
*
* Copyright (C) 2009 Ruslan Savchenko
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
*/
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/namei.h>
#include <linux/inotify.h>
#include <linux/sched.h>
#include <linux/string.h>
#define SPY_INOTIFY "[spy_inotify] "
#define MASK (IN_ACCESS | IN_ATTRIB | IN_CLOSE_WRITE | IN_CLOSE_NOWRITE \
| IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_MODIFY \
| IN_MOVE_SELF | IN_MOVED_FROM | IN_MOVED_TO | IN_OPEN )
static char *path = "/";
module_param(path, charp, 0);
MODULE_PARM_DESC(path, "Path to spy at");
static char comm[TASK_COMM_LEN];
void sinotify_event(struct inotify_watch *watch, u32 wd, u32 mask,
u32 cookie, const char *name, struct inode *inode);
void sinotify_destroy(struct inotify_watch *watch);
static struct inotify_operations si_op = {
sinotify_event, sinotify_destroy };
void sinotify_event(struct inotify_watch *watch, u32 wd, u32 mask,
u32 cookie, const char *name, struct inode *inode) {
task_lock(current);
strncpy(comm, current->comm, sizeof(current->comm));
task_unlock(current);
printk(KERN_INFO SPY_INOTIFY
"Catched inotify event for file %s. mask=%d, pid=%d executable=\"%s\"\n",
path, mask, current->tgid, comm);
}
static struct inotify_handle *ih = 0;
static struct inotify_watch watch;
void sinotify_destroy(struct inotify_watch *watch) {
}
int init_sinotify(void)
{
struct path s_path;
int err;
printk(KERN_INFO SPY_INOTIFY "Spying at %s\n", path);
if ((err = kern_path(path, LOOKUP_FOLLOW, &s_path)) != 0) {
printk(KERN_ERR SPY_INOTIFY "kern_path() returned %d\n", err);
goto out;
}
/* FIXME: crappy nullcheck and no actions on error. Odd */
if ((s_path.dentry == NULL) || (s_path.dentry->d_inode == NULL)) {
printk(KERN_ERR SPY_INOTIFY "NULL dentry or inode\n");
err = -ENOENT;
goto out;
}
ih = inotify_init(&si_op);
if (IS_ERR(ih)) {
printk(KERN_ERR SPY_INOTIFY "inotify_init() returned bad ptr\n");
err = PTR_ERR(ih);
goto out;
}
inotify_init_watch(&watch);
err = inotify_add_watch(ih, &watch, s_path.dentry->d_inode, MASK);
if (err < 0) {
printk(KERN_ERR SPY_INOTIFY
"inotify_add_watch() returned bad descriptor: %d\n", err);
goto clean_ih;
}
return 0;
clean_ih:
inotify_destroy(ih);
out:
return err;
}
void exit_sinotify(void)
{
inotify_rm_watch(ih, &watch);
inotify_destroy(ih);
printk(KERN_INFO SPY_INOTIFY "End spying\n");
}
module_init(init_sinotify);
module_exit(exit_sinotify);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Ruslan Savchenko");
MODULE_DESCRIPTION("Spy-mode inotify");
Makefile
obj-m += spy_inotify.o all: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules clean: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
Friday, October 9, 2009
.pathrc
For a long time I've been using a 'usr' directory in my home to install additional/patched software built from scratch. It is very convenient since the main distribution keeps clean and using of environment variables makes it look like everything is normal. For some unknown reason long in the past i've moved variable initializations out of my .bashrc file to .pathrc which is presented below.
Hehe, .pathrc is not something I'm very proud about but several days ago I suddenly urged for it without having any access to my home computer. So let it be posted here.
Hehe, .pathrc is not something I'm very proud about but several days ago I suddenly urged for it without having any access to my home computer. So let it be posted here.
# set PATH so it includes user's private bin if it exists
USER_PREFIX=~/usr
if [ -d ${USER_PREFIX}/bin ] ; then
PATH=${USER_PREFIX}/bin${PATH:+:$PATH}
export PATH
fi
if [ -d ${USER_PREFIX}/sbin ] ; then
PATH=${USER_PREFIX}/sbin${PATH:+:$PATH}
export PATH
fi
if [ -d ${USER_PREFIX}/include ] ; then
C_INCLUDE_PATH=${USER_PREFIX}/include${C_INCLUDE_PATH:+:$C_INCLUDE_PATH}
export C_INCLUDE_PATH
fi
if [ -d ${USER_PREFIX}/include ] ; then
CPATH=${USER_PREFIX}/include${CPATH:+:$CPATH}
export CPATH
fi
for l in lib lib64; do
if [ -d ${USER_PREFIX}/${l} ] ; then
LIBRARY_PATH=${USER_PREFIX}/${l}${LIBRARY_PATH:+:$LIBRARY_PATH}
export LIBRARY_PATH
fi
if [ -d ${USER_PREFIX}/${l} ] ; then
LD_LIBRARY_PATH=${USER_PREFIX}/${l}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export LD_LIBRARY_PATH
fi
done
if [ -d ${USER_PREFIX}/share/man ] ; then
# MANPATH should be ended with ":", otherwise man(1) doesn't
# search in system paths. See manpath(1) for details.
MANPATH=${USER_PREFIX}/share/man:${MANPATH}
export MANPATH
fi
Tuesday, October 6, 2009
Writing a DMA Bootloader
Preamble
At work I was asked to write a program which performs a dma transfer. After reading some documentation, articles and posts on the internet i've done it finally.Since I haven't found a tutorial for this I'll try to write one here.
Abstract
Bootloader sets up the environment for the OS kernel, loads the kernel and transfers execution to it. By using DMA mode the job of loading the kernel and other jobs can be done in parallel thus reducing boot time. Here I'll try to present a DMA mode transfer as early as possible.
Master Boot Record
First of all what is a bootloader? On startup x86 processor executes instruction at address 0xFFFFFFF0 ([3], 9.1.4 "First Instruction Executed") which is located in BIOS area ([2], 3.1.3 "BIOS Memory"). After BIOS finishes hardware checking and other jobs it loads first sector (512 bytes), called Master Boot Record, of a boot device into 0x0000:0x7c00 memory location, saves drive number into DL register and transfers execution control to 0x7c00 ([1], 6.5.1 "Booting from BAIDs"). Although it is very well known that MBR must be ended with 55 AA signature I can't find exact document proclaiming that. So bootloader is just a 510-byte length code with additional 0xaa55 (little-endian) after the end to be executed in real mode at addres 0x7c00.
It is possible to use BIOS fuctions in bootloader. They are invoked as interrupts using 'int' instruction ([3], 6 "Interrupt and Exception Handling"; [3], 16.1.4 "Interrupt and Exception Handling" in real mode; see [5] for detailed description of 'int' instruction), parameters are passed through registers (AH is typically used for selecting a function while interrupt vector selects a function class). The large reference known as Ralf Brown's Interrupt List is available on the internet (see Links). For example, to print a message one can use the following function (in NASM syntax; see Links for NASM manual):
BIOS can even be used for loading a sector from hdd to memory. For more details refer to Rallf Brown's Interrupt List.
Here addrpacket is a data structure stored in the main memory. In the following example addrpacket tells BIOS to transfer the second sector of the disk drive to address 0x7e00 (or DS:0x7e00, RBIL doesn't say it explicitly)
It is up to BIOS to choose which mode the latter transfer is performed in. It could be DMA as well as PIO so we can't rely on BIOS here.
ISA DMA vs PCI IDE DMA
The idea of DMA is older than PCI. On the internet there are plenty of tutorials and examples describing ISA DMA (see Links). If you read something about two DMA controllers (8237A compatible, in particular) or I/O ports 0x00 - 0x0f, then the talk is about ISA DMA. What we need here is IDE DMA which is quite more complicated.
PCI and PCI Configuration Space
In x86 architecture there are two address spaces supported by the processor: main memory ([4], 3.3 "Memory Organization") and I/O address space which elements are called ports ([4], 13 "Input/Output"). While accessing main memory is done by 'mov' and several other instructions, I/O ports are accessed by 'in' and 'out' instructions (see [5], [6] for detailed description of these instructions)
PCI presented another address space called "PCI Configurations Space" in which each PCI device obtains a 256-byte area. PCI Configuration Space is accessed using two ports: 0x0cf8 and 0x0cfc ([8], 3.2.2.3.2 "Software Generation of Configuration Transactions"). The first one is called CONFIG_ADDRESS and the second one is called CONFIG_DATA. Configuration transaction works as follows: software writes to CONFIG_ADDRESS to specify which device's configuration register should be accessed via CONFIG_DATA. Then it reads or writes CONFIG_DATA port thus performing load to or store from device's configuration space. CONFIG_DATA is 32-bit port and 4 bytes aligned at 4-byte boundary are accessed at once.
CONFIG_ADDRESS has the following format ([8], 3.2.2.3.2 "Software Generation of Configuration Transactions")
Enable bit specifies if CONFIG_DATA will be accessed. Function Number selects a function in multi-functional device - PCI device which has separate PCI configuration space for each supported function number. For example, PIIX controller we consider later here has PCI-ISA bridge as function 0 and PCI IDE controller as function 1.
PCI Configuration space for each PCI device consists of header and device dependent region ([8], 6.1 "Configuration Space Organization"). Header is specified as follows
Here we only use Vendor ID (0x00), Status (0x06), Class Code (0x09) and BAR4 (0x20). When talking about Class Code it is often split into three bytes: base class code (0x0b), subclass code (0x0a) and programming interface (0x09) ([8], 6.2.1 "Device Identification"). Base Address Registers are used to specify mapping between either main memory or i/o space and device memory ([8], 6.2.5 "Base Addresses"). Bit 0 of BAR determines the type of memory: 0 for main memory, 1 for i/o space.
For more information see [10], 24 "The PCI Local Bus" and [8].
PCI IDE contorller
To peform DMA IDE transfer IDE controller must be capable of Bus Master function. Such device is determined as follows: base class code and subclass code must both be 0x01 to identify IDE device, and bit 7 of programming interface must be 1 to identify that the device is capable of Bus Master operation ([7], 5.0. "PCI Specifics"; [8], Appendix D "Class Codes"). PIIX/PIIX3 controller is a multi-functional PCI device and it's function 1 is exactly an IDE device with class code 0x010180 ([2], 2.3. "PCI Configuration Registers—IDE Interface (Function 1)", 2.3.6. "CLASSC - CLASS CODE REGISTER (Function 1)").
We need to use 2 bits of PCI Command register ([2], 2.3.3. "PCICMD - COMMAND REGISTER (Function 1)"):
ATA ports are called Legacy IDE ports and are used for sending ATA commands. They have fixed addresses, we'll talk about them when discussing ATA part of DMA protocol. Bus Master ports have changeable base address which is configurable via BAR4.
BAR4 is called Bus Master Interface Base Address Register (BMIBA) in PIIX/PIIX3 documentation ([2], 2.3.9. "BMIBA—BUS MASTER INTERFACE BASE ADDRESS REGISTER (Function 1)"). Bits 2-15 specify 4-bytes aligned base address, bit 0 is hardwired to 1 to indicate this BAR is for I/O address space, other bits are reserved and set to 0.
The latter I/O address range is called PCI Bus Master IDE Registers ([2], 2.7. "PCI BUS Master IDE Registers") and includes the following ports:
Command and Status ports are 8-bit registers while Descriptor Table Pointer is 32-bit one. Bit 3 of Command Register is used for specifying Read/Write operation (Bus Master Read operation reads from memory, Bus Master Write operation writes to memory), bit 0 is used for Start/Stop Bus Master operation. Bit 0 of Status Register indicates if Bus Master operation is in progress, bit 1 indicates errors. For more details refer to [2], 2.7. "PCI BUS Master IDE Registers".
Bits 2-31 of Descriptor Table Pointer Register specify a 4-bytes aligned base address for Physical Region Descriptor Table. PRD Table entry has the following format ([7] 1.2. "Physical Region Descriptor"; [2], 3.5.3. "BUS MASTER FUNCTION"):
Region Address is an address in main memory associated with DMA transfer. Byte Count specify the latter region size in bytes. EOT bit indicates if this PRD is the last one in the table. Note that Physical Region must not cross 64K boundary.
ATA
Legacy IDE I/O ports have constant addresses ([2], 3.5.1. "ATA REGISTER BLOCK DECODE"). They are divided into four blocks: Primary Command Block (base 0x01f0), Primary Control Block (base 03F4h), Secondary Command Block (base 0x0170), Secondary Control Block (base 0x0374). Here registers of command block are presented ([9], 6.2 "I/O register descriptions")
Each register is one byte length. Bit 7 of Status register is called BSY and indicates that device holds control block registers. If it is set all writes to control block registers are ignored. Bit 6 of Status register is called DRDY and indicates that device is capable of accepting a command.
To perform a DMA transfer software must do the following steps ([9], 8.6 "DMA data transfer commands")
a) The host reads the Status or Alternate Status register until the BSY bit is equal to 0;
b) The host writes the Device/Head register with the appropriate DEV bit value;
c) The host reads the Status or Alternate Status register until the BSY bit is equal to 0 and the DRDY bit is equal to 1;
d) The host writes any required command parameters to the Features, Sector Count, Sector Number,
Cylinder High, Cylinder Low and Device/Head registers;
e) The host initializes the DMA channel;
f) The host writes the command code (0xc8 for DMA Read with retry) to the Command register;
Conclusion
Given specifications are enough to perform a DMA transaction. Code with large number of comments is given below.
References
[1] BIOS Boot Specification, Version 1.01, January 1996, Compaq, Phoenix, Intel
[2] 82371FB (PIIX) AND 82371SB (PIIX3) PCI ISA IDE XCELERATOR, April 1997, Intel
[3] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A: System Programming Guide, Part 1, June 2009, Intel Corporation
[4] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 1: Basic Architecture, June 2009, Intel Corporation
[5] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2B: Instruction Set Reference, A-M, June 2009, Intel Corporation
[6] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2B: Instruction Set Reference, N-Z, June 2009, Intel Corporation
[7] Programming Interface for Bus Master IDE Controller, Revision 1.0, 5/16/94, Brad Hosler, Intel Corporation
[8] PCI Local Bus Specification, Revision 2.3, March 29, 2002, PCI Special Interest Group
[9] Information Technology - AT Attachment Interface with Extensions - 2 (ATA-2), Working Draft, Revision 4c, March 18, 1996, X3T10
[10] The Indispensable PC Hardware Book, Third Edition, Hans-Peter Messmer, ADDISON-WESLEY
Links
http://wiki.osdev.org/MBR_(x86)
http://wiki.osdev.org/Memory_Map_(x86)
http://wiki.osdev.org/PCI
http://wiki.osdev.org/BIOS
http://wiki.osdev.org/RBIL
http://wiki.osdev.org/ISA_DMA
http://www.nasm.us/doc - NASM manual
http://www.ata-atapi.com/hiwchs.html - CHS to LBA translation
http://www.bswd.com/cornucop.htm - links to ata/atapi specfications drafts
http://www.cs.cmu.edu/~ralf/files.html - BIOS interrupts
http://www.osdever.net/tutorials/lba.php - an example of data transfer in PIO mode
http://www.nondot.org/sabre/os/articles/TheBootProcess - articles about booting and bootloader examples
http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/dma.html - ISA DMA explanation
The Code
This code works under qemu. Just save all files, run make and execute 'qemu -hda hdd' command. You should see a message "Hello world from DMA Bootloader" indicating everithing goes well.
loader.asm
sector.asm
Makefile
Known Issues
If BIOS sets DS to nonzero value the code fails. Add initializing DS to zero at the very start to ensure the code is correct.
This code doesn't check for _any_ errors. If you want to add error checks refer to corresponding documentation.
No DMA timing configuration is done. Doing this can improve transaction speed significantly. UDMA Mode 5 is 6 times faster than UDMA Mode 0.
On DMA transfer completion controller sends an interrupt. This also should be taken care of.
After issuing a DMA transfer command to device one would probably want to wait until device is ready. In this case add the following code before setting Start/Stop bit of Bus Master Command Register. But be careful and read [9] before for this standard doesn't enfocrce setting DRQ here
At work I was asked to write a program which performs a dma transfer. After reading some documentation, articles and posts on the internet i've done it finally.Since I haven't found a tutorial for this I'll try to write one here.
Abstract
Bootloader sets up the environment for the OS kernel, loads the kernel and transfers execution to it. By using DMA mode the job of loading the kernel and other jobs can be done in parallel thus reducing boot time. Here I'll try to present a DMA mode transfer as early as possible.
Master Boot Record
First of all what is a bootloader? On startup x86 processor executes instruction at address 0xFFFFFFF0 ([3], 9.1.4 "First Instruction Executed") which is located in BIOS area ([2], 3.1.3 "BIOS Memory"). After BIOS finishes hardware checking and other jobs it loads first sector (512 bytes), called Master Boot Record, of a boot device into 0x0000:0x7c00 memory location, saves drive number into DL register and transfers execution control to 0x7c00 ([1], 6.5.1 "Booting from BAIDs"). Although it is very well known that MBR must be ended with 55 AA signature I can't find exact document proclaiming that. So bootloader is just a 510-byte length code with additional 0xaa55 (little-endian) after the end to be executed in real mode at addres 0x7c00.
It is possible to use BIOS fuctions in bootloader. They are invoked as interrupts using 'int' instruction ([3], 6 "Interrupt and Exception Handling"; [3], 16.1.4 "Interrupt and Exception Handling" in real mode; see [5] for detailed description of 'int' instruction), parameters are passed through registers (AH is typically used for selecting a function while interrupt vector selects a function class). The large reference known as Ralf Brown's Interrupt List is available on the internet (see Links). For example, to print a message one can use the following function (in NASM syntax; see Links for NASM manual):
; prints a null-terminated string pointed by DS:SI
print: lodsb
test al, al
je .done
mov ah, 0x0e ; BIOS int 0x10/AH=0x0e function. AL = character to write
mov bx, 0x0007 ; BH = page number, BL = foreground color
int 0x10 ; display a character on the screen, advancing the cursor
; and scrolling the screen as necessary
jmp print
.done: ret
BIOS can even be used for loading a sector from hdd to memory. For more details refer to Rallf Brown's Interrupt List.
; read from disk drive in LBA mode
; disk drive specified by DL is be used
mov si, addrpacket
mov ah, 0x42 ; BIOS int 0x13/AH=0x42 function. DL = drive number
; DS:SI = disk address packet address
int 0x13 ; performs READ from disk drive in LBA mode
Here addrpacket is a data structure stored in the main memory. In the following example addrpacket tells BIOS to transfer the second sector of the disk drive to address 0x7e00 (or DS:0x7e00, RBIL doesn't say it explicitly)
addrpacket db 0x10 ; size of this disk address packet structure
db 0 ; reserved
dw 1 ; number of blocks to transfer
dd 0x7e00 ; transfer buffer in main memory
dd 1, 0 ; starting absolute 64-bit block number
It is up to BIOS to choose which mode the latter transfer is performed in. It could be DMA as well as PIO so we can't rely on BIOS here.
ISA DMA vs PCI IDE DMA
The idea of DMA is older than PCI. On the internet there are plenty of tutorials and examples describing ISA DMA (see Links). If you read something about two DMA controllers (8237A compatible, in particular) or I/O ports 0x00 - 0x0f, then the talk is about ISA DMA. What we need here is IDE DMA which is quite more complicated.
PCI and PCI Configuration Space
In x86 architecture there are two address spaces supported by the processor: main memory ([4], 3.3 "Memory Organization") and I/O address space which elements are called ports ([4], 13 "Input/Output"). While accessing main memory is done by 'mov' and several other instructions, I/O ports are accessed by 'in' and 'out' instructions (see [5], [6] for detailed description of these instructions)
PCI presented another address space called "PCI Configurations Space" in which each PCI device obtains a 256-byte area. PCI Configuration Space is accessed using two ports: 0x0cf8 and 0x0cfc ([8], 3.2.2.3.2 "Software Generation of Configuration Transactions"). The first one is called CONFIG_ADDRESS and the second one is called CONFIG_DATA. Configuration transaction works as follows: software writes to CONFIG_ADDRESS to specify which device's configuration register should be accessed via CONFIG_DATA. Then it reads or writes CONFIG_DATA port thus performing load to or store from device's configuration space. CONFIG_DATA is 32-bit port and 4 bytes aligned at 4-byte boundary are accessed at once.
CONFIG_ADDRESS has the following format ([8], 3.2.2.3.2 "Software Generation of Configuration Transactions")
31 30 24 23 16 15 11 10 8 7 2 1 0
+-+----------+------------+---------------+-----------------+-----------------+-+-+
| | Reserved | Bus Number | Device Number | Function Number | Register Number |0|0|
+-+----------+------------+---------------+-----------------+-----------------+-+-+
31 - Enable bit
Enable bit specifies if CONFIG_DATA will be accessed. Function Number selects a function in multi-functional device - PCI device which has separate PCI configuration space for each supported function number. For example, PIIX controller we consider later here has PCI-ISA bridge as function 0 and PCI IDE controller as function 1.
PCI Configuration space for each PCI device consists of header and device dependent region ([8], 6.1 "Configuration Space Organization"). Header is specified as follows
+----------+-------------+-------------+---------------+----------------------+
| register | bits 31-24 | bits 23-16 | bits 15-8 | bits 7-0 |
+----------+-------------+-------------+---------------+----------------------+
| 00 | Device ID | Vendor ID |
| 04 | Status | Command |
| 08 | Class code | Revision ID |
| 0C | BIST | Header type | Latency Timer | Cache Line Size |
| 10 | Base address #0 (BAR0) |
| 14 | Base address #1 (BAR1) |
| 18 | Base address #2 (BAR2) |
| 1C | Base address #3 (BAR3) |
| 20 | Base address #4 (BAR4) |
| 24 | Base address #5 (BAR5) |
| 28 | Cardbus CIS Pointer |
| 2C | Subsystem ID | Subsystem Vendor ID |
| 30 | Expansion ROM base address |
| 34 | Reserved | Capabilities Pointer |
| 38 | Reserved |
| 3C | Max latency | Min Grant | Interrupt PIN | Interrupt Line |
+----------+-------------+-------------+---------------+----------------------+
Here we only use Vendor ID (0x00), Status (0x06), Class Code (0x09) and BAR4 (0x20). When talking about Class Code it is often split into three bytes: base class code (0x0b), subclass code (0x0a) and programming interface (0x09) ([8], 6.2.1 "Device Identification"). Base Address Registers are used to specify mapping between either main memory or i/o space and device memory ([8], 6.2.5 "Base Addresses"). Bit 0 of BAR determines the type of memory: 0 for main memory, 1 for i/o space.
For more information see [10], 24 "The PCI Local Bus" and [8].
PCI IDE contorller
To peform DMA IDE transfer IDE controller must be capable of Bus Master function. Such device is determined as follows: base class code and subclass code must both be 0x01 to identify IDE device, and bit 7 of programming interface must be 1 to identify that the device is capable of Bus Master operation ([7], 5.0. "PCI Specifics"; [8], Appendix D "Class Codes"). PIIX/PIIX3 controller is a multi-functional PCI device and it's function 1 is exactly an IDE device with class code 0x010180 ([2], 2.3. "PCI Configuration Registers—IDE Interface (Function 1)", 2.3.6. "CLASSC - CLASS CODE REGISTER (Function 1)").
We need to use 2 bits of PCI Command register ([2], 2.3.3. "PCICMD - COMMAND REGISTER (Function 1)"):
PCI Command Register (address 0x04-0x05)
+-----+------------------------------------------------------------------------+
| bit | Description |
+-----+------------------------------------------------------------------------+
| 2 | Bus Master Enable (BME). 1 - enable, 0 - disable |
| 0 | I/O Space Enable (IOSE). 1 - enable access to ATA and Bus Master ports |
+-----+------------------------------------------------------------------------+
ATA ports are called Legacy IDE ports and are used for sending ATA commands. They have fixed addresses, we'll talk about them when discussing ATA part of DMA protocol. Bus Master ports have changeable base address which is configurable via BAR4.
BAR4 is called Bus Master Interface Base Address Register (BMIBA) in PIIX/PIIX3 documentation ([2], 2.3.9. "BMIBA—BUS MASTER INTERFACE BASE ADDRESS REGISTER (Function 1)"). Bits 2-15 specify 4-bytes aligned base address, bit 0 is hardwired to 1 to indicate this BAR is for I/O address space, other bits are reserved and set to 0.
The latter I/O address range is called PCI Bus Master IDE Registers ([2], 2.7. "PCI BUS Master IDE Registers") and includes the following ports:
+--------+--------------------------------------------------------------------+
| Offset | Desscripton |
+--------+--------------------------------------------------------------------+
| 0x00 | Primary Channel Bus Master IDE Command Register |
| 0x02 | Primary Channel Bus Master IDE Status Register |
| 0x04 | Primary Channel Bus Master IDE Descriptor Table Pointer Register |
| 0x08 | Secondary Channel Bus Master IDE Command Register |
| 0x0a | Secondary Channel Bus Master IDE Status Register |
| 0x0c | Secondary Channel Bus Master IDE Descriptor Table Pointer Register |
+--------+--------------------------------------------------------------------+
Command and Status ports are 8-bit registers while Descriptor Table Pointer is 32-bit one. Bit 3 of Command Register is used for specifying Read/Write operation (Bus Master Read operation reads from memory, Bus Master Write operation writes to memory), bit 0 is used for Start/Stop Bus Master operation. Bit 0 of Status Register indicates if Bus Master operation is in progress, bit 1 indicates errors. For more details refer to [2], 2.7. "PCI BUS Master IDE Registers".
Bits 2-31 of Descriptor Table Pointer Register specify a 4-bytes aligned base address for Physical Region Descriptor Table. PRD Table entry has the following format ([7] 1.2. "Physical Region Descriptor"; [2], 3.5.3. "BUS MASTER FUNCTION"):
+---------+----------+----------+----------+----------+
| | byte 3 | byte 2 | byte 1 | byte 0 |
+---------+----------+----------+----------+----------+
| dword 0 | Memory Region Physical Base Address |0|
| dword 1 |EOT| Reserved | Byte Count |0|
+---------+---+------+----------+----------+--------+-+
Region Address is an address in main memory associated with DMA transfer. Byte Count specify the latter region size in bytes. EOT bit indicates if this PRD is the last one in the table. Note that Physical Region must not cross 64K boundary.
ATA
Legacy IDE I/O ports have constant addresses ([2], 3.5.1. "ATA REGISTER BLOCK DECODE"). They are divided into four blocks: Primary Command Block (base 0x01f0), Primary Control Block (base 03F4h), Secondary Command Block (base 0x0170), Secondary Control Block (base 0x0374). Here registers of command block are presented ([9], 6.2 "I/O register descriptions")
+--------+--------------------------------+
| offset | Register Function |
+--------+--------------------------------+
| 0x00 | Data |
| 0x01 | Error / Features |
| 0x02 | Sector Count |
| 0x03 | LBA bits 0-7 |
| 0x04 | LBA bits 8-15 |
| 0x05 | LBA bits 16-23 |
| 0x06 | Device / Head |
| 0x07 | Status / Command |
+--------+--------------------------------+
Each register is one byte length. Bit 7 of Status register is called BSY and indicates that device holds control block registers. If it is set all writes to control block registers are ignored. Bit 6 of Status register is called DRDY and indicates that device is capable of accepting a command.
To perform a DMA transfer software must do the following steps ([9], 8.6 "DMA data transfer commands")
a) The host reads the Status or Alternate Status register until the BSY bit is equal to 0;
b) The host writes the Device/Head register with the appropriate DEV bit value;
c) The host reads the Status or Alternate Status register until the BSY bit is equal to 0 and the DRDY bit is equal to 1;
d) The host writes any required command parameters to the Features, Sector Count, Sector Number,
Cylinder High, Cylinder Low and Device/Head registers;
e) The host initializes the DMA channel;
f) The host writes the command code (0xc8 for DMA Read with retry) to the Command register;
Conclusion
Given specifications are enough to perform a DMA transaction. Code with large number of comments is given below.
References
[1] BIOS Boot Specification, Version 1.01, January 1996, Compaq, Phoenix, Intel
[2] 82371FB (PIIX) AND 82371SB (PIIX3) PCI ISA IDE XCELERATOR, April 1997, Intel
[3] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A: System Programming Guide, Part 1, June 2009, Intel Corporation
[4] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 1: Basic Architecture, June 2009, Intel Corporation
[5] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2B: Instruction Set Reference, A-M, June 2009, Intel Corporation
[6] Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2B: Instruction Set Reference, N-Z, June 2009, Intel Corporation
[7] Programming Interface for Bus Master IDE Controller, Revision 1.0, 5/16/94, Brad Hosler, Intel Corporation
[8] PCI Local Bus Specification, Revision 2.3, March 29, 2002, PCI Special Interest Group
[9] Information Technology - AT Attachment Interface with Extensions - 2 (ATA-2), Working Draft, Revision 4c, March 18, 1996, X3T10
[10] The Indispensable PC Hardware Book, Third Edition, Hans-Peter Messmer, ADDISON-WESLEY
Links
http://wiki.osdev.org/MBR_(x86)
http://wiki.osdev.org/Memory_Map_(x86)
http://wiki.osdev.org/PCI
http://wiki.osdev.org/BIOS
http://wiki.osdev.org/RBIL
http://wiki.osdev.org/ISA_DMA
http://www.nasm.us/doc - NASM manual
http://www.ata-atapi.com/hiwchs.html - CHS to LBA translation
http://www.bswd.com/cornucop.htm - links to ata/atapi specfications drafts
http://www.cs.cmu.edu/~ralf/files.html - BIOS interrupts
http://www.osdever.net/tutorials/lba.php - an example of data transfer in PIO mode
http://www.nondot.org/sabre/os/articles/TheBootProcess - articles about booting and bootloader examples
http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/dma.html - ISA DMA explanation
The Code
This code works under qemu. Just save all files, run make and execute 'qemu -hda hdd' command. You should see a message "Hello world from DMA Bootloader" indicating everithing goes well.
loader.asm
;
; DMA Bootloader
;
; This program is provided as an example for educational
; purpose ONLY and WITHOUT ANY WARRANTY
;
; Copyright 2009, Ruslan Savchenko
;------------------------------------------------
; macroses
;------------------------------------------------
%macro PCI_CONF_READ 1-2
; %1 - pci address register base
; %2 - pci conf register num
mov eax, %1
%ifnum %2
or ax, %2
%endif
call pci_conf_read
%endm
%macro PCI_CONF_WRITE 1-2
; WARNING! mangles ebx!
; %1 - pci address register base
; %2 - pci conf register num
mov ebx, %1
%ifnum %2
or bx, %2
%endif
call pci_conf_write
%endm
;------------------------------------------------
; consts
;------------------------------------------------
%define iobase 0xff0 ; Bus Master I/O port base
%define sector 57 ; Number of sector to load. Must be less than 256
%define codestart 0x500 ; address where load sector to
;------------------------------------------------
; code
;------------------------------------------------
org 0x7c00 ; tell NASM this code is to be executed at address 0x7c00
start
; configure stack at first
cli ; disable interrupts
mov ax, 0x9000 ; stack segment base
mov ss, ax ; write segment base to SS register
mov sp, 0xffff ; stack pointer
sti ; enable interrupts
; Loop through all PCI devices and their functions
; to find IDE Bus Master
mov cx, 0xffff ; Bus Number, Devce Number and
; Register Number together occupy
; 16 bits of PCI CONFIG_ADDRESS
mov ebx, 0x80000000 ; set Enable bit
.m1
PCI_CONF_READ ebx ; load Device ID and Vendor ID to eax
cmp ax, 0xffff ; check if device is not present
je .m2 ; device not present
PCI_CONF_READ ebx, 0x08 ; load class code, subclass code and
; prog interface to eax
shr eax, 15
cmp eax, 0x010180 >> 7 ; check if class = 0x01,
; subclass = 0x01 and
; bit 7 of prog_if = 1
jne .m2 ; not an IDE Bus Master device
; found Bus Master capable IDE controller
mov dword [pciide], ebx ; save CONFIG_ADDRESS to access
; IDE Bus Master in future
.m2
add ebx, 0x100 ; try next function/device/bus
loop .m1
mov eax, iobase | 1 ; Bus Master Interface Base Address
PCI_CONF_WRITE [pciide], 0x20 ; save address to BMIBA Register
PCI_CONF_READ [pciide], 0x04 ; PCI Command register
or ax, 1 ; set I/O Space Enable Bit
PCI_CONF_WRITE [pciide], 0x04 ; save PCICMD
; wait while ATA device is bisy
mov dx, 0x01f7 ; ATA Primary Status/Command
.m3
in al, dx
cmp al, 0x80 ; check if BSY bit is set
jae .m3
; for Primary device set use of Drive 0 (Master) in LBA mode
mov dx, 0x01f6 ; ATA Primary Device/Head
mov al, 0x40 ; LBA mode, Drive = 0, LBA 27-24 = 0
out dx, al
; wait for ATA defice to accept command
mov dx, 0x01f7 ; ATA Primary Status/Command
.m4
in al, dx
and al, 0xc0 ; BSY and DRDY bits
cmp al, 0x40 ; check if BSY=0, DRDY=1
jne .m4
; Set parameters of transaction:
; nubmer of sectors, first sector LBA address and mode
;
;mov dx, 0x01f2 ; ATA Primary Sector Count
;mov al, 1 ; count = 1
;out dx, al
;
;mov dx, 0x01f3 ; ATA Primary LBA 0-7
;mov al, sector ; LBA 0-7 = sector
;out dx, al
;
;mov dx, 0x01f4 ; ATA Primary LBA 8-15
;mov al, 0 ; LBA 8-15 = 0
;out dx, al
;
;mov dx, 0x01f5 ; ATA Primary LBA 16-23
;mov al, 0 ; LBA 16-23 = 0
;out dx, al
;
;mov dx, 0x01f6 ; ATA Primary Device/Head
;mov al, 0x40 ; LBA mode, Drive= 0
;out dx, al
;
; Using loop here reduce code size by 12 bytes
mov cx, 5 ; 5 iterations
mov dx, 0x01f2 ; ATA Primary Sector Count
mov si, atasector ; transaction parameters array
cld ; string operations will increase SI
.m5
lodsb ; load next byte to al and increase SI
out dx, al ; write byte to corresponding ATA Primary
; Command Block port
inc dx ; increase DX
loop .m5
; initialize host dma transfer
PCI_CONF_READ [pciide], 0x04 ; PCI Command Register
or ax, 4 ; enable Bus Master Function
PCI_CONF_WRITE [pciide], 0x04 ; save PCICMD
; set Drive 0 DMA Capable in Bus Master IDE Status Register
; This bit does not affect hardware operation
;mov word dx, iobase | 0x2 ; Bus Master IDE Status Register
;in al, dx
;or al, 0x20 ; set Drive 0 DMA Capable
;out dx, al ; save Bus Master IDE Status
; write PRD Table pointer
xor eax, eax
mov ax, prd
mov dx, iobase | 0x4 ; Bus Master IDE Descriptor
; Table Pointer Register
out dx, eax
; command ATA to perform DMA transfer
mov dx, 0x01f7 ; ATA Primary Status/Command
mov al, 0xc8 ; DMA Read With Retry Command
out dx, al
; tell Bus Master to start DMA transfer
mov dx, iobase ; Bus Master IDE Command Register
mov al, 9 ; Start/Stop Bit set
; Read/Write Bit set
out dx, al
;
; Place to do some other job while sector is transfered from
; drive to memory without holding CPU
;
; wait until transfer is finished
mov dx, iobase | 0x2 ; Bus Master IDE Status Register
.m6
in al, dx
bt ax, 0 ; Bus Master IDE Active Bit
jc .m6
; transfer execution to recently loaded code
jmp (codestart >> 4):0
;------------------------------------------------
; functions
;------------------------------------------------
pci_conf_read
; Input:
; EAX = PCI CONFIG_ADDRESS
; Output:
; EAX = PCI CONFIG_DATA
mov dx, 0xcf8 ; CONFIG_ADDRESS
out dx, eax
mov dx, 0xcfc ; CONFIG_DATA
in eax, dx
ret
pci_conf_write
; Input:
; EAX = DATA
; EBX = PCI CONFIG_ADDRESS
; Output:
; EBX = DATA
xchg eax, ebx
mov dx, 0xcf8 ; CONFIG_ADDRESS
out dx, eax
mov dx, 0xcfc ; CONFIG_DATA
mov eax, ebx
out dx, eax
ret
;------------------------------------------------
; data
;------------------------------------------------
; array of bytes to be passed through ATA I/O ports
atasector db 1, sector, 0, 0, 0x40
; sector count = 1
; LBA 0-7 = sector
; LBA 8-15 = 0
; LBA 16-24 = 0
; device/head = 0x40 (LBA mode, drive 0)
align 4
; Physical Region Descrptor
prd dd codestart ; Memory Region
dd 0x80000200 ; EOT, 512 bytes
; buffer to save CONFIG_ADDRESS of PCI IDE device
pciide dd 0
;------------------------------------------------
; Space for Partition Table
;------------------------------------------------
times 436-($-$$) db 0xff
;------------------------------------------------
; boot sector magic end
;------------------------------------------------
times 510-($-$$) db 0
dw 0xAA55 ; MBR Signature
sector.asm
;------------------------------------------------
; code
;------------------------------------------------
org 0x500 ; tell NASM to assume this program
; begins at 0x500 when loaded into memory
start:
mov si, hellomsg
call print_si
cli ; disable interrupts
hlt ; infinite loop which can be broken only by interrupt
;------------------------------------------------
; functions
;------------------------------------------------
print_si
push ax
push bx
.next
lodsb
test al, al
je .done
mov ah, 0x0e ; BIOS int 0x10/AH=0x0e function. AL = character to write
mov bx, 0x0007 ; BH = page number, BL = foreground color
int 0x10 ; display a character on the screen, advancing the cursor
; and scrolling the screen as necessary
jmp .next
.done:
pop bx
pop ax
ret
;------------------------------------------------
; data
;------------------------------------------------
hellomsg db 'Hello World from DMA Bootloader', 13, 10, 0
;------------------------------------------------
; sector end
;------------------------------------------------
times 512 -($-$$) db 0
Makefile
hdd: loader sector
rm -f hdd
dd if=loader of=hdd bs=512 count=1
dd if=sector of=hdd bs=512 count=1 seek=57
loader: loader.asm
nasm loader.asm
sector: sector.asm
nasm sector.asm
clean:
rm -f hdd loader sector
Known Issues
If BIOS sets DS to nonzero value the code fails. Add initializing DS to zero at the very start to ensure the code is correct.
This code doesn't check for _any_ errors. If you want to add error checks refer to corresponding documentation.
No DMA timing configuration is done. Doing this can improve transaction speed significantly. UDMA Mode 5 is 6 times faster than UDMA Mode 0.
On DMA transfer completion controller sends an interrupt. This also should be taken care of.
After issuing a DMA transfer command to device one would probably want to wait until device is ready. In this case add the following code before setting Start/Stop bit of Bus Master Command Register. But be careful and read [9] before for this standard doesn't enfocrce setting DRQ here
; wait until ATA device is ready to transfer data
mov dx, 0x01f7 ; ATA Primary Status/Command
.m7
in al, dx
test al, 0x08 ; check if DRQ bit is set
jz .m7
Subscribe to:
Posts (Atom)
